DOCUiBB? BBS08B 



IB 001 055 

Dick, Robert A.; And Others 

Personnel Technology! Using an Inexpensive 

Computer- Based System to Teach ferforaance-Oriented 

Skill. Technical Beport Bo. 1728-1. Pinal Report. 

Hasan Pactors Research, Inc., Goleta, Calif. 

office of Waval Research, Washington, D.C. Personnel 

and Training Research Progress Office. 

HFR-TR-1728-1 

1 Jan 74 

166p.; March 27, 1972— Decesber 31, 1973 (Report 
period) 

HP-10.75 HC-17.80 PLUS POSTAGE 

♦Computer Assisted Instruction; ^Computer Science 

Education; Blectronic Technicians; Perfornance Based 

Education; *Prograaed Instruction; *Prograaed Texts; 

♦Simulation 
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while the students in the text-based curriculum used 
diagrams of digital logic circuits and written explana- 
tions in lieu of demonstrations and troubleshooting 
exercises. 



Prior to training, students took the Navy's General 
Classification Test (verbal aptitude) and Electronic 
Technician Selection Test (electronics and mathematics 
aptitude). Members of matched pairs on the aptitude 
tests were randomly assigned to the two training treat- 
ments . 

Two performance criteria were used to measure training 
effectiveness; one was a multiple-choice written test 
of theoretical knowledge, and the second was a one-hour 
session of troubleshooting actual faulty digital logic 
circuits. Both performance tests were administered 
,twice--once within a week after the student completed 
the 15-hour course, and again nine weeks after completion 
of the course. 

Several different analyses were used to relate student 
performance on the written and skill tests to the two 
aptitude measures and to short-term and long-term 
retention. Observation and anecdotal evidence suggested 
that the computer-based curriculum was more interesting 
and stimulating, especially to the brighter students; 
however, none of the objective measures showed an 
advantage for the computer group. In fact, there 
appeared to be a slight superiority of the programmed 
text-based curriculum for the students of lower verbal 
and/or electronics and math aptitude on the long-term 
retention test in troubleshooting. 
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I. Executive summary 

Aaong the most severe problems in current Military train- 
ing are (1) the wide, range of trainee aptitudes, (2) the short- 
age of qualified personnel to teach a multiplicity of subject 
matters, and (3) the general scarcity of operational equipments 
available for training practical skills. Many believe that 
computerized instruction offers the potential for solving these 
problems in that, theoretically, (1) computerized instruction 
is flexible enough to provide different instructional strate- 
gies for trainees of differing aptitudes, (2) many standardized 
computerized curricula could be developed that do not require 
qualified classroom instructors, and (3) computerized instruc- 
tion could include simulations of operational equipments. 

One of £he major obstacles to wide implementation of com- 
puterized instruction is cost. The primary costs are (1) 
hardware and software acquisition and maintenance, and (2) 
-professional educators 1 time in modifying existing courses, 
or creating new ones, to fit the logical constraints of the 
computerized approach. Furthermore, there is not a large 
quantity of objective evidence that the computerized approach 
is the best of several alternate instructional strategies.. 
Many comparisons between computerized and "conventional" 
instruction use an unmodified classroom/lecturer approach as 
the "conventional" basis for comparison. The results of such 
a comparison confound the benefits of modifying the course 
with the benefits of using a computer. 

The objectives of the present research were (1) to in- 
vestigate the feasibility of cutting costs in computerized 
instruction by limiting the computer to necessary roles (e.g., 
providing simulations of operational equipment, controlling 
the instructional environment, but not disseminating textual 
information, which can be done less expensively with a textbook); 
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(2) to objectively compare this computerized curriculum with 
a non-computerized curriculum that contained identical course 
information in an identical logical format; 43} to make the 
results of the study relevant to skill training by using a 
subject matter, troubleshooting digital logic circuits, that 
emphasized learning a practical skill; and (4) to relate the 
results to the problem of differing trainee aptitudes. 

Two experimental curricula were developed that met the 
criteria within the objectives outlined above. One used a 
minicomputer, the other used programmed instruction. The cur- 
ricula were administered to trainees who were matched on scores 
on two Navy aptitude tests, the General Classification Test and 
the Electronic Technician Selection Test. After the courses, 
trainees were tested for knowledge via a paper-and-pencil 
test, and for skill via a 1-hour timed test in troubleshooting 
digital logic circuits. Nine weeks after the courses, knowl- 
edge and skill tests were again administered to the trainees. 

The results showed that high aptitude students were 
superior to low aptitude students on all the post-cour»e per- 
formance critexia, whether the criteria were measures of 
acquired knowledge or task performance. The training treat- 
ment had nb differential effect on the high aptitude students. 
Low aptitude students in the non-computer curriculum appeared 
to perform better on certain skill criteria than did low apti- 
tude students in the computerized curriculum. Students in 
the non-computerized curriculum finished the training sooner 
than those in the computerized curriculum, regardless of apti- 
tude level. We concluded from the results of the present 
study, and from certain relevant literature, that (1) high 
aptitude students arc relatively insensitive to different 
training curricula; (2) low aptitudu students are, indeed, 
sensitive to different training curricula; (3) computerized 
instruction is not necessarily faster; (4) more, not less, in- 
structor involvement may be the case in computerized instruction, 
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especially at the outset when students are learning to inter- 
act with the system; and (5) evidence is accumulating that 
predicts loss expensive computerized instruction in the future. 

We recommended that (1) further research on computerized 
instruction include adequate control curricula for comparison; 
(2) research on optimizing instruction for individuals of 
varying aptitude should focus on the lower aptitude trainees; 
and (3) alternative, less expensive, instructional approaches 
should be carefully considered before opting for computerized 
instruction in skill training. 



II. INTRODUCTION 



The research reported here was motivated by the problems 
of training skills in military organizations. We experimentally 
investigated several aspects of the applicability of computer- 
ized instruction to solving some of these training problems. 
The following sections discuss the practical problem background, 
relevant literature, and our approach. 

Background 

Computerized instruction appears to offer the potential 
for alleviating several long-standing problems in civilian and 
military education. Both institutions have had difficulty 
coping with individual differences, but the self-paced in- 
structional sequence, which is, conceptually, at the heart of 
computerized instruction, is a step toward solving the indi- 
vidual differences problem. Both institutions also suffer 
varying degrees of difficulty in providing qualified instruc- 
tors; here, computerized instruction could be of immense value 
in providing standardized, high quality instruction which can 
be distributed to many local educational facilities. A third 
problem, more acute in some military situations, is the absence 
of up-to-date training equipment. Computers afford great ed- 
ucational potential here in that they can be programmed to 
simulate operational equipment. Computerized instructional 
techniques, given appropriate generalized displays and inter- 
facing, can also be brought to bear on this problem. 

Training in the Naval Reserve provides a clear example 
of the potential value of computerized instruction. We use 
the Naval Reserve as an example, because problems in skill 
training, which exist to some degree in all military institu- 
tions, are acutely present in the Reserve. The individual 
differences of Naval Reservists in aptitude, rate, level of 



experience, and recency of experience create a strong require- 
ment for self-paced, individualized instruction. The extreme 
heterogeneity of the personnel in technical specialities makes 
it highly improbable that qualified instructors will be available, 
in sufficient quantity and at the right places, to meet the 
need. In addition, the high cost of military hardware makes 
it prohibitive to provide up-to-date equipment and training 
devices to the large number of Reserve Training Centers 
scattered throughout the country. 

An indication of the multiplicity and magnitude of the 
Reserve training problem is the fact that there are 132,000 
authorized Naval Reserve pay billets in more than 400 Naval 
Reserve Training Centers spread across the United States. 
Each training center may, at any given time, be required to 
provide training for personnel from each of the more than 60 
Navy rates. Only rarely is it possible to provide training 
for a particular pay grade within a rate and almost no con- 
sideration can be given to providing training suitable for, a 
particular NEC. Confronted with this problem, Naval Reserve 
training relies heavily on repeated use of the same written 
course materials and a very limited amount of "hands-on" train- 
ing, often with very outmoded equipment. The consequences are 
not only inadequate training for rapidly changing fleet re- 
quirements but very limited motivation as well. 

It is tempting to look to computerized instruction to 
solve training problems of the type posed by the Naval Reserve. 
Unfortunately, many computerized instructional developments 
are associated with large computer systems requiring costly 
initial hardware and software development investments as well 
as expensive, continuing hardware and software maintenance. 
Two important questions may be raised here: "Does computerized 
instruction deliver the benefits that have been hypothesized?" 
And, if so, "are there means of reducing the cost of computerized 



instruction to bring 4t within the grasp of institutions with 
modest training budgets?" 

Relevant Literature 

Two general areas of research are relevant to the ques- 
tions stated above. One is concerned with the problem of 
individual differences in training, the other, naturally, with 
the field of computerized instruction itself. 

1. The Problem of Individual Differences in Training 

4 

The hypothesis that instruction should be tailored to meet 
the needs of individuals with differing aptitudes, motivation, 
and other characteristics is intuitively appealing. The basic 
premise is that not all individuals will maximally benefit 
from a single kind of instructional treatment; for instance, 
high aptitude students may do better in an instructional en- 
vironment that allows a great deal of autonomy than they would 
in an instructional environment that., is very rigid and direc- 
tive, while the reverse may be true for low aptitude students. 
Cronbach (1957) provided one of the early discussions of this 
hypothesis. This area of research has been defined as the 
Aptitude/Treatment Interaction (ATI) problem. 

^Bracht (1969) analyzed 90 research studies on ATI. He 
argued that the only interaction between instructional treat- 
ment and student aptitude that was practically significant 
would be an interaction such as the one described above where 
one treatment was superior for students at one level of apti- 
tude but inferior for students at another level of aptitude. 
Bracht called this a "disordinal" interaction. An interaction 
of this sort would be important to discover, because it would 
indicate the need for "individualized" training; different 
training treatments for different aptitude levels. 



Bracht contrasted the above type of interaction with what 
he called an "ordinal" interaction where one of two treatments 
nay be superior for both aptitude levels; however, the dif- 
ference between the two treatments may be much smaller for one 
aptitude level than for the other aptitude level. This latter 
type of interaction may be statistically interesting; however, 
it would not indicate the need for individualized training; a 
single training treatment would be best for all aptitude levels. 

Bracht found only five studies that showed disordinal inter- 
actions and he concluded that in these cases the findings had 
no clear implications for individualized instruction. Glaser 
and Resnick (1972) discussed the ATI problem and, alon& with 
Bracht, concluded that few or no ATI effects had been solidly 
demonstrated. Glaser and Resnick concluded that the "...nega- 
tive results raised significant questions about this area of 
research." 

The following studies, which were not included in the 
above reviews, suggest that this area of research is hot a 
dead topic. These three studies point to a number of practical 
problems in training students of high and low aptitude. 

Bialek et at. (1973) reported some very interesting work 
exploring the best way to train Army men of differing apti- 
tudes. They were especially concerned with Level IV trainees 
(those scoring between 10 and 30 on the Armed Forces Qualifica- 
tion Test), but they also explored the middle and upper apti- 
tude ranges. This rather large research program on ATI began 
with a relatively rigorous factorial approach through systema- 
tic manipulation of various aptitude, media, and subject matter 
variables. These investigators eventually concluded that this 
scientific and systematic approach became too unwieldy and 
expensive. They then switched to what they called an "optimi- 
zation strategy" where they hypothesised the "best" training 
strategy for teaching low aptitude students and a different 
"best" training strategy for high aptitude students. For both 
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the high and the low aptitude strategies , they proceeded 
through successive iterations of tryout and modification in 
order to converge upon "optimal training strategics" for each 
aptitude level. Briefly, Bialek arrived at the following con- 
clusions: (1) Low aptitude students do poorly in experimental 
studies because they do not appear to sec the relevance of the 
learning task and therefore are not well motivated; (2) low 
aptitude students do best under conditions that maximize inter- 
action with a live instructor; (3) low aptitude students do not 
do well with printed material (e.g., programmed text); and (4) 
high aptitude students do best under conditions that permit 
autonomy and they do not appear to need costly audio-visual 
equipment and similar materials. 

McFann (1971) emphasized the need for research on training 
personnel of high and low aptitudes in military organizations, 
especially the Army, by noting that "an important difference 
between Army training and public education is that the Army 
must utilize the products of its instructional system." Work- 
ing on the same project as Bialek, McFann reported that an 
associate, Grimsley, made the interesting finding that for 
medium and high ability trainees, the fidelity of a training 
simulator, with respect to the actual operational equipment, 
could be very low with no adverse effect on the time needed 
for training, the level of proficiency, the amount remembered 
over time, or the time needed for retraining. However, the 
fidelity of the training device was found to be very important, 
in the training of low ability personnel. For the low ability 
group, the higher the fidelity of the device, the greater the 
proficiency achieved, and the less time required for training. 

Dick and Latta (1969) reported an interesting result in 
training two non-overlapping groups of low and high math apti- 
tude junior high level students in learning certain mathematical 
concepts. These investigators compared paper-and-pencil pro- 
grammed text instruction with computer-aided instruction that 
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used a CRT to display text material. The type of instructional 
method made little difference in the proficiency achieved by 
the high aptitude students, but the low aptitude students 
achieved higher proficiency with the programmed instruction 
treatment. This phenomenon persisted in the results of a long- 
term retention tost for these students. 

We wish to conclude this very brief overview of the ATI 
problem with the following summation: (1) The ATI problem is 
not nearly a dead issue as suggested by Glaser and Resnick; 
the problems of training students of differing aptitudes are 
very real in practical training situations such as those faced 
by military organizations; (2) further, there is evidence for 
the following generalization: High aptitude students are not 
terribly sensitive to instructional treatments; they appear 
to do well with a wide variety of training methods or media. 
However, low aptitude students are indeed sensitive to the 
training treatment, and the fact that most military organiza- 
tions anticipate an increasing proportion of low aptitude 
personnel makes the problem of finding optimal training 
strategies for low aptitude students a crucial one. 

2. Computerized Instruction 

(a) Training in General, The advent of computer technology 
was seen by many as a possible panacea for the problem of indi- 
vidualizing instruction to meet the needs of students with 
differing aptitudes. Surely the computer with its large storage 
capacity, fast retrieval time, and flexibility in programming 
would be the teacher who could he all things to all men. 

Two broad approaches to utilizing computers in education 
emerged: One was Computer-Assisted Instruction (CAI) where the 
computer, in some sense, replaced the instructor. The other 
approach was Computer-Managed Instruction (CMI) where the com- 
puter became a tool to be used by an instructor as a bookkeeping 
aid in keeping track of his students 1 progress. We will not 
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attempt to review the voluminous literature on CAI and CMI. 
Stoluro (1969), Suppcs et at* (1968), Cooley and Glascr (1969), 
Hansen (1970), and Salisbury (1971), among others, have 
adequately reviewed the CAI literature, extolled their philo- 
sophies for proceeding with CAI research, and described their 
own programs. Other reports include Hickey and Newton's (1967) 
review of the results of the CAI information exchange program 
sponsored by the Office of Naval Research through a contract to 
ENTELEK, Incorporated. Hickey (1970) provides an overview of 
the entire ENTELEK program including the conferences and litera- 
ture surveys conducted under this project. Farr (1972) reviewed 
the CAI research and development programs sponsored by Person- 
nel and Training Research Programs, within the Office of Naval 
Research. 

In addition to general overviews of CAI work, the follow- 
ing two reviews concern specific CAI development programs. 
Dwyer (1971) discusses the NEKBAS I C / CATALYST program developed 
by the University of Pittsburgh. This »is essentially a complex 
software package. Hammond (1972) discusses two advanced hard- 
ware developments in CAI; one is TICCIT (Time-shared Interactive 
Computer Controlled Information Television), which uses tele- 
vision as the primary output display; and PLATO (Programmed 
Logic for Automatic Teaching Operations), which uses a specially 
developed display called the "PLASMA panel" that allows very 
fine resolution. 

The literature cited above is but a small portion of the 
CAI literature and is mainly provided for the interested reader 
who wishes to use it as a jumping-off point for further review. 
A wide range of specific and general conclusions may be drawn 
from these reports. The following points bear on the present 
study. 

There is general concern about the cost of computerized 
instruction. High visibility expenditures are hardware and 
software costs in developing and maintaining CAI and CMI systems. 
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A less conspicuous, but no less avoidable, cost is that of 
professional educators' time in reconstructing old courses, or 
developing new courses, to fit CAI and CMI paradigms. The need 
for course reconstruction , in computerized training has also led 
to problems in comparing the effectiveness of computerized 
training with "conventional" training. The studies that have 
compared the results of computerized instruction with non- 
computerized instruction have often confounded the results 
due to course reorganization with the results due to using a 
computer in the instructional paradigm. An example of this 
will be given in the next section. 

In general, the initial flurry of activity and excitement 
over computerized instruction has settled down. Many investi- 
gators are now turning to the long arduous task of developing 
a sound theory of instruction and implementing it. That im- 
plementation, quite probably, will involve computerized in- 
struction. Before turning from this topic, we wish to point to 
a bright spot on the horizon. Given that a sound theory of 
instruction develops, where computers play some important role, 
it may be possible to implement that theory without the expense 
of the current crop of large-scale CAI systems. Thomas (1973) 
describes the development of a computer-aided instruction system 
using a minicomputer* His cost comparisons for large and small 
computerized instructional systems show comparable costs per 
hour for the large and small systems but a substantial savings 
in initial acquisition for the small system. Brebner (1973) 
describes another computer-assisted instructional system built 
around a relatively small computer, a PDP-8/I. Our own efforts 
along these lines will be described in detail later. 

(b) Skill-Oriented Training. The number of investigators 
who have focused on the problems involved in training practical 
skills is a relatively small proportion of the investigators 
who comprise the fields of education, educational psychology, 
and instructional psychology. Therefore, there has been 
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relatively little computerized instruction work done in these 
fields that bears direct.ly upon skill training; the work that 
has been done was usually funded through contracts with military 
agencies. The following examples are given to characterize 
skill research of this nature. 

Rigney et at. (1972) reported on research and development * 
on practical skill training in a computer-aided environment, 
performed at the Univorsity of Southern California Behavioral 
Technological Laboratories. Briefly, the approach here was to 
have a front panel copy of a unit of operational equipment 
standing alongside a student terminal connected to a computer 
system. A student "conversed" with a software package that 
guided him through exercises in button pushing and knob turn- 
ing on the front panel of the operational equipment. 

There appear to be two main emphases in Rigney 1 s approach: 
The first is to develop a theory and logic for task analysis 
that will apply to a wide variety of specific skills. Much 
of the work involves the breakdown of skills and training prob- 
lems into three elements, "goal descriptions, action descrip- 
tions, and clustering operators," and the delineation of all 
possible relationships between actions and goals. Rigncy's 
second main emphasis has been to develop sophisticated programs 
that are general and embody his approach to task analysis. 
These programs then operate on "data modules" which are the 
simple lists that are specific to a given problem or task area. 
Once Rigney is finished with his work, he would have a logic 
for task analysis and a set of general programs so that, for 
example, an instructor who wished to teach the use of an 
oscilloscope would (1) analyze the task according to a well 
worked out task analysis logic, (2) generate the list for the 
specific data module for teaching this task, (3) plug the 
data module into Rigney 1 s general program, (4) sit the student 
down at a terminal with a copy of the oscilloscope alongside, 
and (5) press "run" on the coinputcrizcr training system. This 
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may be an over-simplification of Rigney's approach; however, 
it appears to characterize the goals of his program. His 
main thrust lies in the philosophy and theory of task analysis 
and the development of sophisticated general-purpose software 
to implement the theory* 

Another group, at the Naval Personnel Research and Develop- 
ment Center (NPRDC) , has- taken a somewhat different tack. In- 
stead of addressing the general problem of skill analysis, as 
Rigncy has done, they plunged ahead with generating one solution 
to a specific practical training problem. Ford et al, (1972) 
reported on the development of a CM approach to training five 
of the eight topics at Navy Basic Electricity/Electronics 
School. They compared CM training with regular classroom 
lecture training and found that the CM approach resulted in 
(1) reduced classroom time (39%-54% less) and (2) better scores 
on the School Examination as well as on a Supplemental Test. 
Significant improvements for the CM groups were evident on 
nearly all criteria, on all topics. They also showed that 
pairs of students working at a single CM terminal performed 
as well as one student per terminal. These investigators per- 
formed several iterations of tryout and revision in order to 
optimize the set of CM curriculums for the five topics. 

Now, Rigney's group and Ford's group represent two very 
different approaches: On the one hand, Rigney's group took 
the somewhat academic approach of studying the theoretical 
parameters behind the problem of computerized skilled training 
and software development. On the other hand, Ford's group 
took the more applied approach of directly generating a solu- 
tion to a training problem using what appeared to be the best 
techniques currently available. At this point, it is not 
clear which approach is better; it is not even clear that one 
can make that sort of value judgment about these two approaches. 
Rigncy has attacked the very difficult problem of developing 
a comprehensive, logical approach to task analysis that may 
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generalize to a wide variety of skill training problems. At 

the sane time, he is developing the means for implementing the 

results of his findings on task analysis* However, Rigney 

has not performed comprehensive comparisons between his ap- 
proach and others. 

Ford's group has made the comparison, and the results of 
their efforts arc currently in use in practical training situa- 
tions; however, this approach also has drawbacks. They have 
attacked a specific set of training problems and their solution 
may not generalize as well as Rigney' s. Also, their large- 
scale comparison may be criticized on the grounds discussed 
earlier: They have confounded the results of course reconstruc- 
tion with the results of using a computerized curriculum. They 
compared the final result of several iterations of CAI curricu- 
lum revision with traditional Navy classroom training; it is 
not at all clear whether the improvement in the CAI condition 
was due to the intensive and extensive course revision or due 
to the use of the computer. 

This latter criticism is reinforced by the results of 
another program at NPRDC. Stern (1972) reports the results of 
a program to develop and evaluate performance-oriented test 
equipment training procedures. He showed that a self-paced 
workbook approach to training the use of test equipment was 
superior to the typical lecture- lab approach in Surface Sonar 
Technician (ST) A-l Training School. Here is an example that 
shows that well-thought-out course reconstruction does not nec- 
essarily need a computer in order to show training results that 
are superior to the typical classroom lecture and laboratory 
approach. K 

3. Summary 

The above review of the literature is by no means exhaus- 
tive. Ke intended simply to outline some of the primary prob- 
lems in the application of computerized instruction to skill 
training. We have shown that a critical problem is the relation- 
ship of an instructional method to the individual characteristics 
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of the trainee. This is particularly important in the case of 
low aptitude trainees. We have focused upon the problems in 
training practical skills and showed that there are pitfalls 
in evaluating the effectiveness of computerized instruction 
in. this area. The next section discusses our experimental 
approach to these problems. 

Research Objectives and Approach 

One objective of this research was to investigate the 
feasibility of cutting computer costs in computerized educa- 
tion by limiting the computer to only those roles that are 
absolutely necessary, i.e., where there is no less expensive 
alternative. For example, many computerized instructional 
systems use the computer to disseminate textual information 
where an obviously cheaper alternative exists, the lowly text- 
book. In this case, the computer is not an absolutely essential 
part of the computerized instructional system. On the other 
hand, the control of the instructional environment, i.e., 
administering tests, evaluating student performance, and pre- 
scribing further instruction, are illustrative of cases where 
the computer may be a cheaper solution than the alternative, 
a teacher. As described in detail later, we developed a 
computerized instructional curriculum that shaved computer 
costs by limiting the computer to only those roles that we 
thought could be performed best by the computer. 

Another objective of this research was to comparatively 
evaluate this computerized curriculum with a non-computerized 
curriculum in order to determine what increased effectiveness, 
if any, resulted from the computerized approach. We went^to 
great extremes to make the comparisons as objective as possible. 
As discussed above, many investigators, in making similar com- 
parisons, have not provided adequate controls for the course 
reconstruction involved in developing the computerized curricu- 
lum. This has made it impossible to assess the effectiveness 
of the computer qua computer against the "conventional" 
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curriculum. In these cases, the comparison is really between 
the conventional curriculum and a vast ly-modified-course-with- 
computer curriculum. The present study avoided confounding 
the effects of course modification in the comparisons of 
computerized and ncn-computcrized instructional approaches. 
We did this by modifying the "conventional" course along the 
same line as the computerized course. 

A further objective of this research was to relate the 
results of this study to the needs of an institution, such as 
the Navy, that has a high interest in training proficiency in 
practical skills. Therefore, we selected a course topic, 
troubleshooting digital logic circuits, that emphasized learn- 
ing a practical skill that is of considerable importance in 
many current Navy training courses. 

A final objective was to relate the effectiveness of the 
two instructional treatments to the problem of training stu- 
dents with different aptitudes. Therefore, two of the inde- 
pendent variables in this study were the level of general 
verbal aptitude and the levftl of science/math aptitude. The 
latter measure of ability is used as one of the important 
predictors of success in Navy courses similar to the course 
in digital logic developed for this study. 
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III. DESCRIPTION OF THE EXPERIMENT 



A. Overview 

As explained in the section on research objectives and 
approach, one of the objectives of this research program was 
to explore the possibility of minimizing costs in computerized 
instruction by limiting the computer to only those roles which 
were essential. Furthermore, we saw this as an opportunity 
to evaluate the contribution of the computer in computerized 
instruction by comparing an alternative non-computerized 
curriculum with a curriculum that utilizes the computer only 
where necessary. We wanted to evaluate the effectiveness of 
each of the two alternatives for students of different apti- 
tudes, and we wanted to use a course topic that contained an 
important skill component so that the results of this study 
would generalize to the larger problem area of skill training. 

1. Course Topic 

In designing a study to meet these objectives, we chose 
a topic that was directly relevant to Navy training. .Simpson 
et alt (1971) reported on a course on digital logic that was 
developed by HFR for Sonar Technicians working on the AN/SQS-26 
sonar system. This course covers the principles that are 
basic to understandiag the new generation of digital devices. 
It was chosen as a vehicle for comparing the two instructional 
approaches in the present study because, intuitively, instruc- 
tion in this topic should benefit from laboratory experience 
with real digital logic circuits. This makes it suitable for 
testing a computerized system that is designed to train skills 
via simulation of operational equipment. Further, a course in 
digital logic is inherently interesting to the Navy because it 
is an important part of many different curricula in the Navy's 
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Class A and C electronics training schools. Finally, because 
HFR developed the digital logic course for tho AN/SQS-26 
Common Core Curriculum, we were in a good position to modify 
and use it for our present purposes. 

2. Non-Computerized Curriculum 

Ke chose Programmed Text Instruction (PTI) as the "con- 
ventional" curriculum to be used as a basis for comparison in 
the present study. This seemed to be an excellent comparator 
for the computerized curriculum because PTI (1) is widely 
used, (2) is relatively inexpensive, and (3) requires the same 
sort of logical course construction as required by a computerized 
curriculum. 

A simple, linear-programmed text format was used because 
of its current use in many instructional contexts; it is a 
convenient and reliable pedagogical vehicle. Because the 
purpose of the present experiment was not to compare different 
PTI formats or modes, we arbitrarily chose one that was con- 
venient and has gained relatively wide acceptance. The only 
danger was that the PTI format might interact with student 
aptitudes and become an unwanted secondary source of varia- 
bility in this study. However, Davis cl> at. (1970) reported 
no ATI in a study of different modes of PTI. They tested 
overt versus covert responding, constructed-response versus 
multiple choice, and feedback versus no feedback. Their null 
ATI results led us to use one of the simplest PTI modes; it 
will be discussed in detail later. 

3. Computerized Curriculum 

The experimental curriculum used a small computer to 
control the learning environment and to drive a display that 
simulated digital logic circuits and allowed students to 
control input to the circuits, test internal circuit condi- 
tions, and observe output conditions. In the computerized 
curriculum, students received the bulk of their needed textual 
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information from a programmed text, just as the PTI students 
did, instead of from a CRT as in many computerized instruc- 
tional systems. 

A computerized instructional system that does not require 
the storage and display of bulky textual information enjoys 
several advantages over one that does. Less storage and much 
less programming are required. A paper or text medium for con- 
veying the bulk of information in a curriculum is cheaply pro- 
duced, reliably stored, does not require expensive displays, is 
easily accessed by the average student, if appropriately in- 
dexed, and is easily edited or modified. And, probably most 
important, the text portion of the curriculum can be produced 
by personnel qualified in the technical area of the course 
without requiring additional expertise in computer programming. 
Clearly, a computer is not an absolute necessity for dissemi- 
nating the bulk of curriculum information; in fact, there are 
good reasons not to allocate that function to the computer 
part of a computerized instructional system. 

On the other hand, a valid use of the computer may be to 
control certain of the contingencies in the instructional pro- 
gram; this is particularly true in the context of programmed 
instruction where it may be .desirable to control students 1 
behavior at the many test points. While the mode of programmed 
instruction may not interact with student aptitudes, as pointed 
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out by Davis, the computer may perform a valuable service in 
forcing students to take remediation when they appear to need 
it. Much of computerised instruction is really computer- 
presented programmed instruction. Therefore, we included this 
aspect of computerized instruction in order to test the possi- 
ble advantage of computer control of the training environment. 
The only reasonable alternative here, close tutorial supervision 
by a live instructor, is too costly. 

Another viable use of the computer, and one for which 
there is no inexpensive alternative, is the simulation of 
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operational equipment. Here the computer can drive a labora- 
tory in which the student may make inputs and observe the 
consequences. The alternative would be a different set of 
real operational equipments for each curriculum and a tutor 
to monitor each student. 

Ke developed a computerized curriculum that filled the 
specifications discussed above. For convenience, we shall 
refer to the computerized curriculum used in the present 
study as a Computer Integrated Instructional (CII) approach; 
we will arbitrarily define CII as the use of a relatively 
small, time-shared computer facility to control certain train- 
ing and testing contingencies for several students working 
simultaneously and independently and to control individual 
displays and response terminals that provide students with 
simulations of operational equipment with which the student 
may interact. Furthermore, the CII approach incorporates a 
linear, frame-oriented, programmed text as the primary vehicle 
for disseminating information to the students. This defini- 
tion of CII is purely arbitrary and is intended merely for 
use in this report as an economic reference to the computerized 
curriculum in the study », opposed to the PTI curriculum. In 
a broader context, CII is a relatively new term in the com- 
puterized instructional literature* but it is still sufficiently 
ambiguous to allow us to customize its definition for our pres- 
ent purposes. 

4. Comparison of the Two Curricula 

As will be explained in detail later, we took great care 
to make the comparison between the CII and the PTI curricula 
fair, viz, wc wanted the content and logical format of both 
curricula to be identical so that the only difference between 
the CII and PTI treatments would be the contribution of the 
computer to the instructional process* At this point, a 
comparison of the two instructional treatments will be made 
in order to complete the overview of exactly what wc were 
trying to accomplish. 
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Table 1 presents a comparison of some of the critical 
elements in the two instructional treatments. It shows that 
students in both curricula received their theoretical in- 
struction in digital logic via hardcopy programmed textbooks; 
it should be emphasized that the textbook material, organiza- 
tion of the material, and format of the material were nearly 
identical for the two instructional curricula. The textbook 
for the PTI curriculum was virtually a^verbatim copy of the 
CII textbook, except that references to the computer system 
were omitted, and diagrams and truth tables were inserted for 
the PTI students where the CII students would look at dia- 
grams on their display and interact with the dynamic circuits. 
These differences are discussed in considerable detail in the 
section on text development. The point emphasized here is 
that a great deal of attention was given to equating the two 
curricula for content, organization, and format. We wanted 
to isolate and reduce the differences between the two curricula 
to only those tha* represent the computer-based instructional 
system's potential advantages, and to test whether those 
advantages would be realized. 

5. Student Aptitudes 

We did not want to concern ourselves with the theoreti- 
cal aspects of ATI, for that could quickly bog down our 
otherwise straightforward experimental interests. We felt, 
however, as a practical matter, that the results of a compari- 
son between two instructional approaches should at the very 
least be related to the general verbal aptitudes of the students. 
Going one step beyond this, we felt that the results of a com- 
parison between curricula should also be related to students' 
aptitudes that arc most likely related to the course topic. 

Because we wanted to generalize the results of this study 
to the population of Mavy personnel, we chose two N'avy apti- 
tude tests in order to assess general verbal aptitude and 
aptitudes related to the course topic, digital clcctionics. 
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TABLE 1 



COMPARISON OF CRITICAL ASPECTS OF THE PROGRAMMED TEXT 
INSTRUCTION (PTI) CURRICULUM AND THE COMPUTER INTEGRATED 
INSTRUCTION ( CI I ) CURRICULUM IN DIGITAL LOGIC 



PTI 


CII 


Hard-copy programmed textbook 
on theory in digital logic. 


Hard-copy question frames 
interspersed among information frames. 


Student conceives his 
responses to questions. 


Student types his responses 
to questions. 


Hard-copy feedback frames 
follow question frames. 


Computer controlled display 
of feedback after student 
responds . 


Errors result in a branch back to an 
Information frame, or a branch forward to 
remediation. Eventually, the question is repeated. 


Student may ignore the 
branch and the suggested 
exercise; he may proceed 
without correctly answer- 
ing the question. 


Student cannot Ignore the 
branch and he must per- 
form the suggested exer- 
cises; he cannot proceed 
without correctly answer- 
ing the question. 


Hard-copy demonstration 
frames utilize diagrams 
and truth tables to 
demonstrate digital 
logic principles and 
circuits. 


Computer-driven general 
displays utilize plastic 
overlay diagrams to pre- 
sent dynamic demonstra- 
tions of digital logic 
principles and operating 
circuits. Students inter- 
act with system. 


Hard-copy troubleshooting 
frames utilize diagrams 
and truth tables to simu- 
late faulty circuits. 
Student locates fault. 


Computer-driven displays 
utilize plastic overlay 
diagrams to present 
dynamic simulations of 
faulty circuits. Stu- 
dents interact with sys- 
tem to test circuit com- 
ponents and locate faults. 



TABLE 1 (Continued) 



PTI 


CII 


Student errors in troubleshooting 
result in a branch back to a prior 
information frame or demonstration, 
or a branch forward to remediation. 
Eventually, the troubl eshooti ng exerci se is repeated. 


Student may ignore the 
branch and suggested 
exercise; he may proceed 
without correctly trouble- 
shooting the faulty cir- 
cuit. > 


Student cannot ignore 
the branch and he must 
perform the suggested 
exercise; he cannot 
proceed without cor- 
rectly troubleshooting 
the faulty circuit. 
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The two Navy tests were the General Classification Test (GCT) 

« 

and the Electronic Technician Selection Test (ETST) . The 
latter is, in reality, a science background test with fairly 
heavy loadings in math and electronics. Hereafter, we will 
refer to scores on the ETST as reflecting "science/math apti- 
tude"; and scores on the GCT as reflecting "verbal aptitude." 

6. Performance Criteria 

After a student has completed a training course, the 
question of importance is: "Can he perform the operational 
task?" While the most obvious method of answering this ques- 
tion is to confront the student with the operational task, it 
is often not possible or convenient to do this. Therefore, 
paper and pencil "knowledge" tests are widely used as surro- 
gates for an actual test of performance. In the present study, 
we used one such surrogate criterion, a multiple-choice Knowl- 
edge Test. We also used a performance test that was very nearly 
like the operational task of interest. This was a Skill Test 
that required each student to troubleshoot faulty digital logic 
circuits under the observation of a test proctor who scored his 
errors and measured the time required to identify the various 
faults. 

It is of obvious operational importance that the training 
also produces retention of skills; in many operational contexts, 
the opportunity to exercise learned skills and to receive 
added reinforcement is limited or sporadic. Therefore, we 
administered the Knowledge and Skill Tests both at the end of 
training and again nine weeks later in order to assess the 
longer-term effects of the two curriculn. 

The following sections discuss the experimental plan, 
the details of the curricula development, and the results. 

Experimental Plan 

Figure 1 shows a flow chart of the general experi mentn 1 
plan. Each of the activities in Figure 1 will be discussed 



26 



ADMINISTER APTITUOE 
SELECTION TESTS TO 
POTENTIAL STUDENTS 



RANDOMLY 
STUDENTS 
APTITUDE 
TRAINING 



DISTRIBUTE 
MATCHED ON 
TESTS TO TWO 
CURRICULA 



KNOWLEDGE 
PRETEST 



COMPUTER 
INTEGRATED 
INSTRUCTION 
15-HOURS 



PROGRAMMED 
TEXT 

INSTRUCTION 
15-HOURS 



1 






FINAL 






TESTS 




2-MONTH 


INTERVAL 




RETENTION 






TESTS 





Figure 1. General experimental plan. 



in considerable detail in later sections of this report; here 
we wish to briefly summarize each activity so that the reader 
may keep in mind the overall plan as he progresses through the 
somewhat detailed material ahead. 

The two aptitude tests were administered to a population 
of volunteers for the study at the Federal Correctional Insti- 
tution at Lompoc, California. The characteristics of this 
population also will be discussed later. 

The scores of the selection or aptitude tests were ana- 
lyzed in order to create matched pairs of students who were 
equated on both verbal aptitude and science/math aptitude. 
The members of each pair then were randomly distributed, one 
to the CII curriculum and one to the PTI curriculum. 

Prior to the beginning of the training sessions, all 

IT 

students were administered a knowledge pretest to provide 
something of a base line for measuring the effects of training. 
Then they were given five 3-hour training-sessions in digital 
logic theory and faulty logic circuit troubleshooting. Half 
of the students received the CII curriculum and half received 
the PTI curriculum. The training sessions were conducted in 
small group sessions that included four CII students and four 
PTI students in the classroom at any given time. The room was 
divided in half, and one proctor monitored both groups. 

After two such groups of students completed the training 
sessions, they received two final tests, the paper and pencil 
Knowledge Test and the Skill Test involving the troubleshooting 
of faulty digital logic circuits in specially-created test 
equipment. No feedback was given for the Knowledge Test but 
feedback was inherent to the Skill Test. After an interval of 
2 months, the students again were given the Knowledge Test, 
which was identical to the one previously taken, and a new 
Skill Test, which required troubleshooting of circuits identi- 
cal to those used in the prior performance tests but with 
different faults. 
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IV. DETAILED PROCEDURES AND EQUIPMENT DEVELOPMENT 



Development of Experimental Curricula 
1. General Approach 

We have stated that our intent was to produce two experi- 
mental curricula that would be as equivalent as possible with 
the exception of *hc variables of experimental interest. Fig- 
ure 2 shows the overall project plan. The development of the 
experimental curricula and the necessary support material is 
highlighted in this project plan. The critical path is shown 
with the bold arrows. 

Because the CI I approach had the most stringent format 
requirement, the specifications for the CI I curriculum and 
related text were developed first. 

The next step was the development of the specifications 
for the necessary software and hardware modifications to an 
existing computerized instructional system that HFR had pre- 
viously developed for other research (Mccherikof f , 1974). 
Subsequent steps on the critical path included (1) execution 
of the specified modifications, (2) writing the computer code 
to control the CI I training environment, and (3) pilot work to 
check out the system and the curriculum code. 

After the curriculum for the CII students was relatively 
stable, a modification of that curriculum was made to meet the 
needs of the PTI approach. Basically, the PTI curriculum was 
generated by (1) excising all references to the computer in 
the CII text and (2) substituting diagrams, truth tables, and 
short discussions at the points in the CII curriculum where 
CII students would interact with the laboratory simulations. 

Other activities in the development of the experimental 
curricula included the development of displays for the Cll 
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curriculum, performance criteria, and special equipment for 
the skill tests. 

2. Text 

A linear programmed text was the basic vehicle for both 
curricula. Question frames were interspersed among informa- 
tion frames; questions were multiple choice with from two to 
four alternatives. An incorrect answer resulted in a branch 
back to an information frame or a branch to a separate section 
containing remedial information frames. Correct answers re- 
sulted in the student going on to the next information frame. 
In addition to information frames, special demonstration 
frames were constructed. In these, specific circuits were 
used as vehicles to demonstrate the principles that had been 
discussed in information frames. In the PTI curriculum, the 
demonstration frames used explanations, truth tables, and 
line-drawing diagrams to demonstrate the operation of various 
circuits. In the CII curriculum, the demonstrations used 
explanations in the text, simulated circuits on the displays 
at the student terminal, and student interaction with the 
system. His inputs were interpreted by the computing system 
and would result in state changes at various points in the 
digital logic circuits on the display. 

There was also a special kind of question frame called a 
Troubleshooting Exercise. Here, a circuit with a fault, was 
presented and the student's task was to identify the faulty 
component in the circuit or the aspect of the circuit's logic 
which was faulty. For the PTI students, faults were indicated 
by errors in the truth table associated with the circuit. For 
the CII student, faults were indicated by the display showing 
erroneous logic states at various points in the circuit. A 
student's performance in troubleshooting was evaluated by his 
answer to a question that immediately followed each trouble- 
shooting exercise. Again, a multiple-choice format was used 
with an incorrect response resulting in a repeat of a 
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demonstration or information frame. Eventually, the student 
had to repeat the troubleshooting exercise. Correct answers 
allowed the student to proceed. 

The CII students were completely captured by the system; 
they were not allowed to proceed without correctly answering 
questions. Errors forced the student to go back to the 
appropriate demonstration, repeat the demonstration, return 
to the question, and correctly answer it. However, the PTI 
students, if they were so inclined, could ignore the branching 
suggestion and the exercise repeats. 

The text was divided into six volumes; this was done in 
order not to intimidate the students when they began the course. 
Although the course could be completed easily within the 15-hour 
time limit, the text comprised a rather imposing mass of mate- 
rial. The following list of lessons will give the reader some 
idea of the material covered in the curriculum: 

Volume Lesson 'No. Lesson Title 

I 1 Introduction 

II 2 OR Gate 

3 AND Gate 

III 4 Inverters and the 

NOR and NAND Gates 

5 Equivalent Logic Functions 

IV 6 Combination Logic Functions 

7 R-S Flip-Flop 

V 8 Clocked R-S Flip-Flop 

9 D Flip-Flop 

10 J-K Flip-Flop 

VI 11 Shift Hegister 

12 Ring Counter 

13 Divider 

14 Multiplexers 
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It should be evident that this curriculum takes the stu- 
dent from the very basics of digital logic through most of 
the common topics and finally to a fairly sophisticated under- 
standing of digital logic in the context of ring counters, 
dividers, and multiplexers. At first, we were worried that 
it might not be possible to adequately present this material 
within the 15-hour time frame; however, pilot work with sev- 
eral students showed that persons of average intelligence 
could work through the programmed text well within our desired 
time limit. We felt that it was necessary to present a fairly 
extensive curriculum in order to tax the abilities of the 
brighter students and to provide a large base of knowledge to 
serve as a vehicle for our later tests. On the other hand, 
we did not want to expend a great deal of costly subject and 
experimenter time on a much longer curriculum. Given these 
competing factors, a 15-hour course, comprising the range of 
sophistication represented by the topics listed above, 
appeared to be a good compromise. 

Appendix A contains an excerpt from Lesson 5, "R-S Flip- 
Flop," for the PTI text. Appendix B contains an excerpt of 
the same material from the CII text. The reader is invited 
to compare the two excerpts to see the similarities and 
differences between the CII and the PTI texts, and to get a 
general feeling for the digital logic course that was developed. 

3. CII Terminals 

Figure 3 shows a student sitting at a CII terminal; the 
terminal comprises a keyboard, a 16-light message display, a 
3-digit display, and a "lab display" for presenting the digi- 
tal logic circuits. The student read his text at the terminal 
and interacted with the system at points indicated in his 
text. Figure 4 shows a close-up of the terminal; the lab 
display uses plastic overlays imprinted with the various 
digital logic circuits. A matrix of light-emitting diodes 
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beneath the overlays was driven by the computer system to 
indicate the logical states (on « "true," off » "false") at 
various points in the logic circuits. For troubleshooting 
exercises, a probe was used to test logic states at different 
points in a "faulty" logic circuit. The probe made contact 
with metallic buttons behind the plastic overlays. The com- 
puter controls the state values for these buttons. 

The overlay in Figure 4 shows the circuit for an R-S 
Flip-Flop, and it is part of the CII curriculum for the text 
excerpt comprising Appendix B. An annotation in Appendix B 
will indicate to the reader where the overlay in Figure 4 is 
used by the student. 

We felt it desirable to minimize the mechanical aspects 
of a student's interaction with the system. Obviously, a con- 
siderable amount of non-pedagogically oriented interaction 
with the computer system would unfairly disfavor the CII 
approach when compared with the PTI approach in a closely 
controlled experiment. Also, we did not want to make the CII 
curriculum an exercise in typing skill. Therefore, typing at 
the terminal was minimized by making most responses a series 
of one or two key pushes. We even assigned one special key, 
an "«0" value, and another special key, an "si" value, so 
that the student would not have to hit two keys for an input 
specification to a logic circuit. The reader may wish to look 
again at Appendix B where the excerpt from Lesson 5 for the 
CII students shows the response reminders that we provided 
in the left-hand margin of each frame that required inter- 
action with the system. A comparison of those response reminders 
with the keyboard shown in Figure 4 will reveal that very little 
typing skill was necessary to interact with the system. 

Budget constraints prevented us from developing really 
general-purpose displays for this project. Because this was 
a research study, rather than development of a system proto- 
type, it was not really necessary to develop the sort of 
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general display that a vendor for such a system would. The 
needs of many different curricula would dictate a very 
general-purpose display; however, for the present study we 
only required a display sufficiently versatile to represent 
and demonstrate the range of logic circuits in the experimen- 
tal course. The matrix of 32 light-emitting diodes driven by 
the computer and the digital logic circuits printed on plastic 
overlays met this requirement without making immodest inroads 
into our equipment budget. 

4. Computer System and Software 

Figure S shows the Redcor 785 minicomputer, which was the 
heart of the system. It has 12,000 words of core storage, a 
magnetic tape unit, teletype or card input, and considerable 
custom interfacing for a variety of real-time experimental 
environments. In the present study, it was connected to four 
student terminals like the one shown in Figures 3 and 4. 

The Redcor runs under a fully resident, multi-programming 
operating system that is similar in concept to IBM 0/S 360. 
This executive system and curricular programming language that 
uses easy mneumonics were both developed in-house for other 
research programs. They have been well documented by Mecherikoff 
(1974). 

The present study required some minor modifications to 
the interpretive curricular language. Beyond that, the bulk 
of software development for the present study involved writing 
the specific program, written in the curricular language, to 
control the CII environment. 

5. Performance Criteria 

Two kinds of tests were used to evaluate student performance 
after they had completed the course. The first was a paper-and- 
pcncil Knowledge Test that contained 40, four-alternative, 
multiple-choice questions. This test was given twice to the 
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students during the study—subsequent to completing the course 
as a "final" knowledge test and as a follow-up knowledge test 
two months later. By using the same test on both occasions, 
we avoided the problem of reliability and content differences 
between alternate test forms* A copy of the Knowledge Test is 
presented in Appendix C. 

Twenty questions were randomly selected from the Knowledge 
Test for use as a knowledge pretest. No feedback was given to 
students on the pretest, the final test, or the long-term 
retention test. 

The second type of test for evaluating student perfor- 
mance was a Skill Test in troubleshooting digital logic cir- 
cuits. Three special, stand-alone circuits were constructed 
for the troubleshooting tests. They allowed the experimenter 
to introduce a variety of faults into each of the logic cir- 
cuits. Figure 6 shows the three circuit boxes with a master 
power and control box. One of the test circuits is a half- 
adder, another is a multiplexer, and a third is a set of four 
relatively simple combination functions. Figure 7 shows the 
master box connected to the half-adder. In some cases, the 
experimenter wired up the test circuit; in others, the student 
was instructed to make certain connections. Each test circuit 
box contained a hidden set of switches which allowed the 
experimenter to introduce different faults in the circuit. 
Appendix D contains the instructions to students for the 
Skill Tests; these instructions are annotated to show the 
nature of each test fault and an example of an acceptable 
correct answer from the student. 

Upon completion of the course, the Skill Test was admin- 
istered to students individually; this required approximately 
one hour. Students were given problems using each of the 
three test circuits. A problem usually required the student 
to manipulate the circuit, making inputs and testing the re- 
sults, in order to detect and identify a fault. In some 



cases, the student was asked to bring the logic circuit into 
a certain specific state; this required the student to know 
how to make manipulations and to execute" them. Performance 
measures for these tests included (1) the number of errors 
made before completing a specific problem and (2) the total 
time to correctly complete the problem. Only one fault was 
introduced into a circuit for each troubleshooting problem. 
Several different faults were used to create seve.il problems 
for each of the three circuits. There were 14 pre lems in 
all. 

Two months after completion of the course, a second Skill 
Test was given to measure retention. The second Skill Test 
used the same three circuits as did the first; however, dif- 
ferent faults were introduced into each circuit to create 
different problems. This was necessary because of our con- 
cern that the students might remember the correct responses 
to the first set of problems. Because two forms of the 
Skill Test were constructed and administered, it was not 
possible to compare performance on individual items of the 
short-terra and long-term skill tests. However, we did not 
want to lose entirely the capability for such a comparison; 
therefore, we embedded three problems from the first Skill 
Test into the set of problems for the second Skill Test. 
Students taking the second Skill Test did not appear to real- 
ize that three of the problems were the same as three problems 
in the first Skill Test*. 

Because our course did not include electronic theory, we 
could not test students directly on the "integrated chips" 
and electronic circuits which underlie digital logic circuits. 
Therefore, our test circuits were constructed so that students 
did not manipulate them at the electronic level, but rather 
at the digital logic level, where symbols wt/re used to replace 
the actual integrated circuits or chips. We pro-wired the 
electronic circuits to appropriate connection points with the 
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symbols. This made it relatively difficult for students to 
make wiring errors that would harm themselves or the circuits. 

'It should be noted, however, that during the test the 
students were really troubleshooting actual logic circuits; 
the only departure from reality was that they did not concern 
themselves with "ground and V cc " connections between the 
integrated circuits and the power supply. Also, because 
they were using symbols, the students did not have to trace 
connections by identifying pins on integrated circuit chips. 
Our purpose was not to test them on circuit tracing, which 
would have necessitated a considerable expansion of the elec- 
tronic aspect of the course. It also would have required a 
good deal more time in the performance test sessions and a 
significant increase in the expenditure of funds in exchange 
for a small increase in the face validity of the performance 
tests. Furthermore, Estelita (1972) has argued that digital 
logic is a viable discipline apart from electronic circuitry. 
He has stated that, "Standardizing. .. IC packages along logical 
function boundaries has in effect divorced the electrical 
circuit designer from the logic designer.... No specific 
technical knowledge is presupposed other than. . .switching 
algebra." Estelita goes on to describe the importance of 
"hands on 11 experience in learning digital logic. His comments 
reinforce the appropriateness of the course topic for the 
current study. 

Some of the problems in the performance tests required 
that students simply recall material that they had learned in 
the course and apply the knowledge and skills for which they 
had received direct training to the solution of a given test 
problem. We called these problems "regular problems." 

Other test problems required more than a simple regurgi- 
tation of knowledge and skills learned in the course; in these 
latter problems, students were confronted with circuits to 
which they had not been exposed before. These problems re- 
quired students to synthesize the basic concepts and skills 



SO 



* 



that had been trained previously into some larger concept of 
digital logic and to use this insight to attack unfamiliar 
problems. We called the latter problems "transfer problems." 
Because this latter sort of activity is an important part of 
applying course knowledge to operational problems, it was felt 
that this was a necessary aspect of comparing the two instruc- 
tional approaches. 

In a word, our intention in devising the Knowledge and 
Skill Tests, especially the skill tests, was to make evaluation 
and comparison of the two instructional approaches generalizable 
and meaningful to practical training situations. This logic 
also led us to include the longer-term retention tests in our 
comparisons of the two instructional approaches. 

B. Field Plan and Execution 

Figure 8 again shows the project plan; in this case the 
activities dealing with execution of the field study are high- 
lighted. These activities included secuiing an appropriate 
source of subjects, selecting the subjects according to verbal 
aptitude and math/science aptitude, administering the two in- 
structional curricula to two groups of subjects matched on 
aptitudes, and, finally, testing subjects upon completion of 
the course and after a long-term retention interval. 

1. Subjects 

Our first target population of potential subjects was 
the student body at Santa Barbara City College; these students 
represented a relatively wide range of verbal and math/sci once 
aptitudes, and they were in the appropriate age range for 
generalizing the results of this study . to Navy trainees. After 
completing arrangements with officials of Santa Barbara City 
College to administer the two experimental curricula at the 
beginning of Fall Quarter, 1973, we di: -uvcred that fewer stu- 
dents would volunteer to participate in the experiment than 
school officials had originally predicted. Various inducements 
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Figure 8. Project plan: Execution of field study 1s highlighted. 

ERIC 



were attempted in order to secure the required number of volun- 
teers so that selection tests could be administered and 100 
students with appropriate characteristics could be selected 
for participation in the courses. An adequate preselection 
population of volunteers could not be obtained; therefore, 
arrangements with Santa Barbara City College were canceled. 

The next most accessible population of subjects with the 
appropriate aptitude and age characteristics resided at the • 
Federal Correctional Institute at Lompoc, California, and at 
the California Men's Colony at San Luis Obispo, California. 
Liaison with these institutions, and the agencies which super- 
vised them, proved to have favorable results in both cases. 
Because the population at the Federal Correctional Institution 
appeared to represent a broader range of verbal aptitude, 
these prisoners were chosen as subjects for the study. 

About 200 inmates volunteered to participate in the 
study. Personnel who completed all aspects of the course 
received a $20.00 stipend and a certificate of course comple- 
tion. The new, short form of the Navy's General Classification 
Test (GCT) and the Electronic Technician Selection Test (ETST) 
were administered to the volunteers to assess verbal aptitude 
and math/science aptitude, respectively. 

A two-dimensional plot of the volunteers' scores on the 
GCT and ETST was made and pairs of nearly-coincident points 
on the plot were identified by inspection. Fifty such matched 
pairs were identified, and the members of a given pair were 
randomly assigned to the two experiment :. 1 treatments. 

A problem that confronted us in selecting subjects for 
this study was that of defining the lowest acceptable aptitude 
level for subjects. We did not want to expend time and funds 
unnecessarily to train and test personnel who clearly could 
not benefit from either of the experimental curricula. Still, 
we wanted to relate the effectiveness of each of the curricula 
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to varying levels of aptitude, including the lowest level 
students who could be trained in the relatively sophisticated 
skills in our experimental course. 

We discussed the matter with a senior staff member of the 
Personnel Measurement Research Department of the Navy Personnel 
Research and Development Center at San Diego; he felt that the 
nominal level of verbal aptitude for a Naval training course in 
logic and electronics should be about one standard deviation 
above the general population mean. He also felt that the lower 
limit in verbal aptitude for students who could benefit from 
such a course would be at about the pop lation mean. These 
figures were informal estimates and are not to be construed 
as official Navy opinion. However, we used these estimates 
as guidelines to help reduce study costs by eliminating un-~ ' 
necessary exploration on the lower end of the verbal aptitude 
scale. 

We used a GCT score of 15 as the lower cutoff point on 
verbal aptitude. This is equivalent to 47 on the Navy Stan- 
dard and it is approximately one-third of a standard deviation 
below the Navy Mean of 50. We felt this would ensure that 
our sample would include subjects in the lowest range of 
verbal aptitudes who could reasonably be expected to derive 
some benefit from the experimental training courses. 

We employed no cutoff score for the ETST. We wanted to 
include students representing the entire range of math/science 
aptitudes, given that they met the minimum reasonable require- 
ments for verbal aptitude. 

Some mention should be made of the background of the 
subject population. Since the subjects were prisoners at a 
Federal Correctional Institute (FCI), this may raise some 
questions in the reader's mind about the general i zabi 1 ity of 
the results to other populations. We did not feel that this 
was a particular problem in this study because of the general 
characteristics of the inmates, some of which will be described. 
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The nominal age range for inmates at the Lompoc FCI is 18 
to 26 years. Their criminal background was varied: a large 
proportion had violated Federal drug laws, another large pro- 
portion included Vietnam war dissenters, still another fairly 
large proportion included convicted bank robbers, and a fourth 
group included Indians who had committed crimes on Federal 
Reservations. Except for th5 last group, the inmates appeared 
to represent a fairly broad range of socioeconomic backgrounds. 
The entire range of educational class levels was represented, 
including adequate representation of the upper educational 
levels, especially by the war dissenters. Also, a selection 
factor was operating because wo dealt only with inmates who 
volunteered; those who participated in the study demonstrated 
relatively high motivation, intelligence, and interest in the 
subject matter. In fact, HFR staff members involved in train- 
ing or testing the subjects felt that, in most cases, they 
were dealing with the sorts of people who might have been 
volunteers from the original target population at Santa 
Barbara City College. 

In short, the students at the prison behaved very much 
as students anywhere. The main differences between the prison 
environment and a "normal" environment were (1) the restric- 
tive atmosphere of the institution itself and (2) an under- 
current of verbal hostility directed toward the institution. 
These two factors might well exist to some degree in a train- 
ing course at a military institution. We could find nothing 
in the range of aptitudes, or in the biographical material 
that we collected for this population, that argued against 
generalizing the results of this study to trainees in a skill- 
oriented curriculum at a military institution or a vocational 
school . 

2. Training and Testing 

Students were trained in groups of 8: 4 PT1 students and 4 
CI I students were supervised by one proctor who was knowledgeable 
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in the course subject, the experimental design, and operation 

• * 

of the CII software and hardware. Later, performance tests 
were administered by a different proctor who had no knowledge 
of a student's aptitudes or instructional treatment. 

Figure 9 shows the layout of the classroom used for train- 
ing. A visual barrier provided nominal isolation of the PTI 
students from the CII students; however, the proctor was equally 
accessible to both groups. The proctor provided assistance 
and explanations whenever requested and he also operated the 
computer system for the CII terminals. The computer and 
basic peripherals were located in a separate room. The com- 
puter required proctor attention only at the beginning of a 
session before students began working at their terminals, 
after a session when students had left, and on infrequent 
occasions during a session when trouble developed with the 
CII system. Otherwise, the proctor was always available to 
assist the students. He kept a covert log of the amount of 
time spent assisting each student and the amount of time 
required by each to complete the course. 



PTI Students 



CII Students 




Figure 9. Layout of classroom. 
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Students were allowed as much time as they wished to 
complete the training phase of the experiment. All of them, 
except two at the lowest aptitude levels, completed the course 
within the 15 hours originally allotted. The two slow stu- 
dents finished after an additional two hours on the Monday 
following the regular 5-day series of 3-hour training sessions. 
Students who finished early were told that they might spend 
time reviewing the material; however, we did not emphasize a 
need to spend a great deal of time in review but allowed them 
to end the training phase as soon as they felt that they 
understood the material. 

The test proctor administered the short-term knowledge 
and performance tests to each student during the week that 
followed his training sessions. This was done in a different 
room so that a new group of students could begin training. 
The test proctor administered the long-term retention tests 
to each student eight weeks after ho had completed the short- 
term retention tests. 

Approximately five months were required to complete the 
training and testing of 106 students. For various reasons, 
several students did not complete all aspects of the training 
and testing program. Complete data were gathered on 42 pairs 
of students matched on the GCT and HTST. Only the data on 
these 84 students, 42 in the CII treatment and M in the PTI 
treatment, were used in subsequent data analyses. 
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V. RESULTS AND ANALYSES 

The independent variables in this study were verbal apti- 
tude (score on GCT) , math/science aptitude (score on ETST) , 
and training treatment (PTI vs. CII). The dependent variables 
were all the measures of knowledge and skill taken at short- 
term and long-term retention intervals, plus two general 
measures of student behavior during training. The following 
is a list of all the dependent variables: 

Measures of Classroom Behavior 

1. Time To Complete Training Course 

2. Number of Interactions with Proctor During 
Training 

Measures of Knowledge 

3. Pre-Course Knowledge Test 

4. Short-Term Knowledge Test 

5. Short-Term Knowledge Gain (4-3) 

6. Long-Term Knowledge Test 

7. Long-Term Knowledge Gain (6-3) 

8. Forgetting (4-6) 
Measures of Short-Term Skill 

9. Time To Complete each of 14 Problems 

10. Total Time To Complete the Skill Test 

11. Number of Timed-Out Problems (where 
arbitrary time limit was exceeded) 

12. Number of Errors on Regular Problems 

13. Number of Errors on Transfer Problems 

14. Total Number of En s on the Skill Test 
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Measures of Long-Term Skill 

15. Time To Complete each of 14 Problems 

16. Total Time To Complete the Skill Test 

17. Number of Timed-Out Problems 

18. Number of Errors on Regular Problems 

19. Number of Errors on Transfer Problems 

20. Total Number of Errors on Skill Test 

Various combinations of the independent and dependent 
variables were used in the three phases of analysis discussed 
below. In the first phase, graphs were used to relate a 
sample of depenoent measures to the independent variables. 
In the second phase, the entire set of dependent measures, 
including item scores on the skill tests, was subjected to a 
series of discriminant analyses. The final phase of the data 
analysis was a series of univariate tests to test the signifi- 
cance of aptitude and training treatment differences with 
respect to single measures. 

Preliminary Data Analysis 

Simple graphs were constructed to show the relationships 
between the aptitude scores and two short-term and long-term 
retention measures. One of the measures was the Knowledge 
Test score; the other was Total Time to Complete the Skill 
Test. The latter measure of skill was used in the preliminary 
analysis, rather than an error measure, because most of the 
students completed all of the problems with relatively few 
errors. Time to Complete the Test appeared to discriminate 
among students better than any of the error measures. Error 
measures were included in later analyses, however. 

Each of the following graphs contains a curve for thje CII 
treatment and a curve for the PTI treatment. The data were 
smoothed by grouping students on the independent variable and 
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deriving group means for the dependent variables. Four figures 
that deal with the Knowledge Test measure are presented on the 
next two facing pages for convenient comparisons. Correlations 
among the variables are presented in Table 2. 

Figure 10 shows the relationship between scores on the 
Short-Term Knowledge Test and scores on the GCT (verbal apti- 
tude) for the PTI and CII training treatments. This figure 
shows the expected relationship between achievement in a 
training course as measured by a written examination, and 
verbal aptitude, viz, the high aptitude students obtained 
higher scores on the Knowledge Test. A score of 18 on the GCT 
is the mean for this test, or the equivalent of a Stanford- 
Binet IQ of 100. Figure 10 shows no systematic mean differ- 
ences in Knowledge Test scores between the C II group and the 
PTI group. The correlation coefficients between GCT and 
Knowledge Test scores were .59 and .65 for the CII and PTI 
groups, respectively. 

Figure 11, like Figure 10, shows performance on the Long- 
Terra Knowledge Test to be positively related to score on the 
GCT; again there were no systematic differences between the 
CII and PTI groups. Correlations between verbal aptitude 
and long-term knowledge were .57 and .47 for the CII and PTI 
groups, respectively. 

Figures 12 and 13 show the corresponding results using 
scores on the ETST (math/science aptitude) as the independent 
variable. It should be mentioned that scores on the GCT and 
the ETST were moderately correlated; the Pearson product- 
moment correlations between GCT and ETST scores for our CII 
and PTI groups were .66 and .68, respectively. Data on the 
population of Navy enlistees who have taken the GCT and ETST 
yield a correlation of about .63. 

Figure 12 shows a strong relationship between performance 
on the Short-Term Knowledge Test and score on the ETST; students 
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Figure 10. Short-term knowledge as a function of verbal 
aptitude and training treatment. 
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Figure 11. Long-term knowledge as a function of verbal 
aptitude and training treatment. 
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Figure 12* Short-term knowledge as a function of math/science 
aptitude and training treatment* 
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Figure 13, Long-term knowledge as a function of math/science 
aptitude and training treatment. 
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with higher aptitudes scored better. There were no system- 
atic mean differences between the CII and PTI groups. The 
correlations between ETST and Short-Term Knowledge Test 
scores for the CII and PTI groups were .74 and .67, respec- 
tively. 

Figure 13 shows score on the Long-Term Knowledge Test 
generally to be an increasing function of score on the ETST; 
there were curious, inexplicable reversals for the highest 
aptitude groups in both the CII and PTI treatments, and for 
the lowest aptitude group in the CII treatment. The number 
of students in these groups at the extreme ends of the apti- 
tude scale was small. Therefore, these reversals probably do 
not represent reliable phenomena. There were no significant 
mean differences between the CII and PTI groups, as tested by 
analysis of variance discussed later. The correlations between 
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ETST and the Long-Term Knowledge Test were .74 and .44 for the 
CII and PTI groups, respectively. This indicates that long- 
term knowledge was less predictable from math/science scores 
for the PTI group than for the CII group. 

The next four figures deal with the Skill Test measure. 

Figure 14 shows the relationship between time required to 
complete the Short-Term Skill Test and scores on the GCT for 
the two training treatments. Time required to complete the 
performance test was a decreasing function of verbal aptitude; 
the brighter students finished the test in a shorter time. 
There appeared to be no systematic mean differences between 
the PTI and CII groups. However, there was a difference in 
the correlations between GCT and Short-Term Skill Test scores 
for the two groups. The correlations were -.52 and -.28 for 
the CII and PTI groups, respectively. Again, aptitude was a 
less potent predictor of performance for the PTI group than 
for the CII group. 

Figure 15 shows that time taken to complete the Long-Term 
Skill Test was a decreasing function of score on the GCT; this 
graph shows a slight superiority for the PTI students, espe- 
cially those of lower verbal aptitude. As discussed later, 
this result was not statistically significant. The correla- 
tions between GCT and Long-Term Skill scores were -.48 and 
-.30 for the CII and PTI groups, respectively. 

Comparison of Figures 14 and 15 reveals a perceptible 
change in the slopes of the curves over the long-term retention 
interval. The relatively flat curves in Figure 15 suggest that 
the lower verbal aptitude students are "catching up" to the 
higher aptitude students in skill performance at the long-term 
test point. However, an analysis of variance, discussed later 
under Univariate Analyses, showed the high aptitude students to 
continue to perform significantly better than the low aptitude 
students at the long-term test point of skill. 
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Figure 14. Short-term troubleshooting performance as a func< 
tion of verbal aptitude and training treatment. 
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Figure 15. Long-term troubleshooting performance as a function 
of verbal aptitude and training treatment. 
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Figure 16. Short-term troubleshooting performance as a func< 
tlon of math/science aptitude and training treatment. 
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Figure 17. Long-term troubleshooting performance as a func< 
tlon of math/science aptitude and training treatment. 
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Figures 16 and 17 show the corresponding results using 
score on the ETST as the independent variable. 

Figure 16 shows that time required to perform the Short- 
Term Skill Test decreased with increasing aptitude as measured 
by the ETST. The two curves cross over and suggest an aptitude- 
treatment interaction; however, statistical tests did not prove 
this interaction to be significant. Correlations between ETST 
and Short-Term Skill Test scores were -.68 and -.38 for the 
CII and PTI groups, respectively. Again, tested performance 
was less predictable from math/science aptitude score for the 
PTI group than for the CII group. 

Figure 17 shows that time required to complete the Long- 
Term Skill Test was also a decreasing function of math/science 
aptitude; furthermore, the low aptitude students in the PTI 
group completed the skill test in significantly less time 
than the low aptitude students in the CII group. The analysis 
to test this result is discussed later. Correlations between 
long-terra skill and math/science aptitude were -.70 and -.34 
for the CII and PTI groups, respectively. Once again, tested 
performance was less predictable from ETST scores for the PTI 
group than for the CII group. 

In summary, the curves in Figure 10 through 17 showed 
the logically expected positive relationship between perfor- 
mance and aptitude. There was also evidence of a possible 
superiority of the low aptitude students in the PTI group over 
the low aptitude students in the CII group on the criterion of 
long-term skill performance. 

It is interesting to note the moderately high correlations 
among jhort- and long-term knowledge and skill tests in Table 
2. Not only arc the correlations high between short- and long- 
term tests of a given kind, but the correlations between kinds 
of tests are also high. In a sense, the va.* iity of the knowl- 
edge test is supported by its high correlations with skill 
measures of high face validity. 
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The next step in the analysis of the data employed a 
multivariate technique to examine the relationships between 
the independent variables and all the dependent variables. 

Multivariate Analysis 

A very powerful multivariate technique exists for con- 
veniently investigating a large number of measures taken on 
individuals that comprise different subpopulations . (The 
subpopulations in the present experiment may be defined as 
the individuals that comprised different aptitude levels in 
the different instructional treatments.) We felt that this 
technique would help us answer the following question: 
"Given all the methods that we used to measure performance, 
were there any real differences between the students from the 
PTI course and those from the CII course?" Furthermore: "If 
there were performance differences, how were they related to 
different aptitude levels?" 

The multivariate technique that we used to address these 
questions is discriminant analysis. Briefly, this technique 
is used when a variety of measures is taken on several pre- 
determined classes of things. The object of the analysis is 
to determine whether or not the classes are different from one 
another with respect to the measures. Discriminant analysis 
is different from factor analysis, another common multivariate 
technique, where a variety of measures is taken upon one popu- 
lation of things with the intention of discovering which sub- 
sets of similar measures can be identified. Detailed dis- 
cussions of discriminant analysis may be found in Rulon (1951A 
and 1951B), Tiedeman (1951), Rao (1952), Cooley and Lohncs 
(1962), and Anderson (1958). 

We performed four separate discriminant analyses: The 
first included all of the dependent variables listed on pages 59 
and 60 except the time to complete each individual item in the 
Skill Tests, and the error scores for the breakdown of "Regular" 
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and "Transfer" items in the Skill Tests. Size limits in tl<e 
discriminant analysis program prevented us from using all the 
dependent measures in the first, general analysis. The second 
discriminant analysis focused on the six measures related to 
the paper-and-pencil knowledge tests. The third analysis 
focused on all the measures of short-term skill; the fourth, 
on all the measures of long-term skill. 

All of these analyses used the same two independent 
variables of instructional treatment (CII versus PTI) and 
score on the ETST (low versus high). Because discriminant 
analysis is rather expensive in terms of computer time, we 
could not thoroughly explore all of the dependent variables 
relative to both of the aptitude variables in this phase of 
analysis. Since the preliminary analysis siiowed possible 
significant* relationships between aptitude and the dependent 
measure in two cases for the ETST and in only one case for 
the GCT, we chose to use the ETST scores as the important 
aptitude variable for discriminant analyses. 

The computer program to perform the discriminant analy- 
sis was Cooley and Lohne's Multiple Discriminant Analysis 
Program, adapted to run on the General Electric Mark III Time 
Sharing System. Before the discriminant analyses were run, 
several small programs were used to pre-process the data into 
manageable 'orm and to sort students into groups according to 
the independent variables of interest. We will not dwell on 
the details of these programs; however, we will describe the 
general algorithm for sorting students and their data. 

Recall that pairs of students matched on GCT and ETST 
scores were identified and members of each pair were randomly 
distributed to the two treatment conditions. In sorting the 
students for the discriminant analyses, we again dealt with 
the matched pairs in the following way: For each matched 
pair, the sum of their scores on the ETST was derived and 
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those sums were ranked. We defined the 16 highest-ranking 
matched pairs (this comprised about 40% of the experimental 
sample) to be the "high aptitude" students, and the 16 lowest- 
ranking matched pairs to be the "low aptitude" students. 
Because each matched pair contained one CII student and one 
PT1 student, we had defined four subpopulat ions , or classes, 
viz, the high aptitude CII students, low aptitude CII students, 
high aptitude PTI students, and finally, low aptitude PTI 
students. We called these the HICOMP, LOCOMP, HiTEXT , and 
LOTEXT groups, respectively. 

These four groups were the "predetermined classes of 
things" which the discriminant analysis program tried to 
discriminate, upon the basis of the dependent variables. To 
the extent that certain measures were differentially related 
to aptitude and treatment levels, the discriminant analyses 
would "separate" the four groups. 

1. General Discriminant Analysis 

The first analysis used 14 general performance measures 
from the list on pages 59 and 60. Item latency scores and er- 
rors for Regular and Transfer problems were excluded. The 14 
measures of performance form the dimensions of a multivariate 
space. To the extent that the individuals in a given group, 
say LOTEXT, perform differently from the individuals in the 
other groups, the swarm of points representing the individuals 
in LOTEXT will occupy a different portion of this 14-space than 
will points representing the other individuals. 

We can reduce this 14-space to something that is concep- 
tually manageable through discriminant analysis on a computer. 
The computer program finds a function that will maximize the 
separation between groups by deriving a set of linear weights 
for the n variables that will minimize the ratio of within- 
group variance to betwecn-group variance. In our geometric 
model, this function would be represented by a line in 14-space. 
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If one projected the points that represent individuals in 14- 
space onto this line, the points for LOTEXT would (hopefully) 
cluster together and be separated from the points for the 
other groups. The points for LOTEXT could be described by 
their mean and standard deviation with respect to this function. 

Having calculated the discriminant function (or, line in 
n-space) that best separates the groups, the step-wise multiple 
discriminant program then finds the next best discriminant 
function for separating the groups. In our geometric model, 
this second line, along with the first, would describe a plane 
in 14-spaco. Again, the points in 14-space could be projected 
onto the plane, and we would expect that this additional dis- 
criminating power would reveal a further separation of the 
projected points for LOTEXT from the projected points for the 
other groups. The points for LOTEXT, when projected upon this 
plane, could be described by their centroid and one standard 
deviation centour. (A centour is a contour that delimits some 
percent of a multidimensional swarm or cloud of points, and 
has as its centroid the centroid of the entire swarm. A one 
standard deviation, or one-sigma, centour would contain 68% 
of the points in a multidimensional swarm.) 

Figure 18 shows the four group centroids and one-sigma 
centours on the plane defined by the first two discriminant 
functions. The first discriminant function, represented by 
the abscissa, accounted for 67.9% of the discriminating power; 
the second discriminant function, represented by the ordinate, 
accounted for 25.8% of the discriminating power. The third 
discriminant function accounted for only 4.64% of the dis- 
criminating power, and therefore no attempt was made to graph 
it on a third axis. 

Inspection of Figure 18 shows that the first discriminant 
function primarily discriminated between the low and hl^h apti- 
tude students. None of the centours for the low aptitude stu- 
dents intersect with the contours for the high aptitude students; 
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this is not a startling result, for our earlier graphs showed 
what common sense would predict--the high aptitude students 
performed better than the low aptitude students. The second 
discriminant function achieved separation between the PTI 
students and the CII students. However, there is some overlap 
between the students for the two treatment groups as evidenced 
by the intersection of the one-sigina centours at both aptitude 
levels . 

Rao's F approximation of Wilke's lambda criterion (Cooley 
and Lohnes) showed that the general discrimination among the 
four groups was significant beyond the .01 level. Basically, 
this test compares the generalized within-group variance with 
the overall variance. 

The positive results of the general discriminant analysis 
indicated the existence of some potent relationships between 
the independent and dependent -variables . The next step was to 
use different sets of measures in subsequent discriminant 
analyses to discover which of the dependent variables were 
importantly related to math/science aptitude and training 
treatment. 

2. Discriminant Analysis Using Knowledge Measures 

The six measures of knowledge listed on page 59 were used 
in the second discriminant analysis. Figure 19 shows the re- 
sults. The first discriminant function accounted for nearly 
90% of the discriminating power of this analysis and it pri- 
marily separated the high and low ap-titude students. The second 
function shows some separation of the treatment conditions at 
the low aptitude level, but this discriminant function only 
accounted for 7,1% of the power in this analysis. 

Rao's F approximation was significant beyond the .01 
level. It appears that the knowledge measure primarily dis- 
criminated aptitude differences. There is weak evidence for 
superior performance of HICOMP over IIITliXT , because the HI TEXT 
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centroid is closer to the centroids for the low aptitude 
groups and HITEXT* s large variance on the first function 
places the bulk of its swarm toward the low aptitude swarms. 
It should be noted that the directions and magnitudes of the 
scales for the discriminant functions do not have simple re- 
lationships with "goodness" of performance; they are a result 
of a complex series of matrix algebra manipulations. The 
figures can be effectively used to discover whether or not 
groups are similar to, or different from, one another with 
respect to a given set of measures. In Figure 19, the HITEXT 
students were more like the low aptitude students on the knowl- 
edge measures than were the HICOMP students. 

3. Discriminant Analysis Using Short-Term Skill Measures 

All the measures listed on page 59 which dealt with per-, 
formance on the Short-Term Skill Test were used in the third 
discriminant analysis. Figure 20 shows the results of tho 
third discriminant analysis. This figure is a little more 
complex than the prior two because it shows the third dis- 
criminant function, which accounted for a sizable amount of 
the power in this analysis. We have drawn the simple ellipses 
for the first two dimensions and then added vectors to show 
where the centroids for each group should be placed along the 
third dimension. We have not attempted to illustrate the 
dispersion of tho swarms in the third dimension; one may 
expect, from the rather low discriminating power of the third 
function, large dispersions relative to the differences in 
centroids. 

Again, the first, and most powerful, discriminant function 
primarily discriminated between aptitude levels. The second 
function showed good discrimination between training treatments 
for the low aptitude students and some discrimination between 
treatments for the high aptitude students. The third function 
showed discrimination between LOTEXT and LOCOMP with respect 
to their centroids. 
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Rao's F-approximation was significant beyond the .01 
level. In general, this analysis showed that the high aptitude <v ^ 
students were fairly similar to one another on the Short-Term 
Skill Test measures, regardless of training treatment, and 
the low aptitude students showed differences related to train- 
ing treatment. Low aptitude students performed differently 
from high aptitude students. 

4. Discriminant Analysis Using Long-Term Skill Measures 

The measures of long-term skill performance listed on 
page 60 were used in the fourth discriminant analysis. Fig- 
ure 21 shows the results plotted in three dimensions. It 
appears that there was little to distinguish the long-term 
skill performance of high aptitude students in the two train- 
ing treatments. In this figure, the centroids for HICOMP and 
HITEXT lie in separate planes defined by the third discrimi- 
nant function. However, this function accounts for only 11.1% 
of the discriminating power; i.e., the separation of centroids 
is sinarll relative to the variance. In essence, the high apti- 
tude students were indistinguishable. 

The low aptitude students in the two training groups were 
well separated by the second discriminant function and, to a 
lesser extent, by the first discriminant function. Their one- 
sigma ccntours showed little overlap. 

Rao's F-approximation was significant beyond the .01 level. 
IVe might conclude from tnis analysis that with respect to the long- 
term retention test of skills, which is probably the criterion 
of highest operational importance, (1) high aptitude students 
performed differently from low aptitude students, (2) training 
treatment did not differentially influence the performance of 
high aptitude students, and (3) training treatment did influence 
the performance of low aptitude students. 
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5. Summary of Results from the Discriminant Analyses 

The results from the series of discriminant analyses may 
be summarized as follows: 

% (a) Aptitude Differences, All of the analyses showed 
large differences in the performance of high and low aptitude 
students. The significant F-tests for generalized discrimina- 
tion were probably due, in large part, to the discrimination 
between students at the different aptitude levels. There is 
no doubt that aptitude differences were generally more impor- 
tant than treatment differences. 

The univariate analyses, discussed next, showed that this 
last statement does not hold for one variable, Time To Complete 
Training. In fact, this variable probably accounted for the 
good separation of training treatments in the first discrimi- 
nant analysis. In the other discriminant analysis, which did 
not include Time To Complete Training, training treatments 
were not so well separated. 

(b) Training Treatment Differences, The only good dis- 
crimination between training treatments at both aptitude 
levels occurred in the first discriminant analysis. As men- 
tioned above, we felt this was primarily due to the Time To 
Complete Training variable. Differences between training 
treatments at the high aptitude level were slight for the 
Knowledge and Short-Term Skill measures and virtually non- 
existent for the Long-Term Skill measures. Differences 
between training treatments at the low aptitude level were 
moderate for the Knowledge measures and considerable for both 
Short-Term and Long-Term Skill measures. 

6. Univariate Analysis 

V.hile discriminant analysis was very useful for exploring 
the maximum possible discrimination between the groups that we 
had defined earlier, it provided no information about the 
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relationships between the independent variables and individual 
dependent variables. To obtain this information we performed 
a series of 18 analyses of variance using all the dependent 
variables that are listed on pages 59 and 60, except the item 
scores for the skill tests. These analyses were performed on 
the same data base used for the discriminant analyses. The 
same four groups, or. cells, were used to represent different 
levels of the independent variables. These were two-by-two 
analyses of variance with two levels of aptitude (high versus 
low ETST scores) and two kinds of troatment (CI I versus PTI). 

Table 3 presents a summary of the results from the 18 
two-by-two analyses of variance. The F-values are presented 
for the treatment effect, the aptitude effect, and their inter- 
action. Asterisks indicate the significance level. It must 
be kept in mind that these variables are not independent of 
one another, and that in a set of significance tests this 
large, some will reach significance purely by chance. We 
performed this set of analyses not in the sense of classical 
hypothesis testing but rather as a means of deriving an index 
of the apparent strength of the relationships between these 
18 dependent variables and the independent variables of 
treatment and aptitude. 

There is a special column in Tabic 3 labeled "Best" Treat- 
ment. This column is to inform the reader which treatment 
condition showed the best performance (e.g., shortest time to 
complete a skill test, highest score on a knowledge test, etc.). 
Entries were made in this column where the treatment effect 
approached/exceeded significance, or where the interaction 
approached/exceeded significance. In all cases where the ap- 
titude effect was significant, the higher aptitude students 
performed "better" than the lower aptitude students, as would 
be expected from our earlier analyses. Inspection of Table 3 
shows that the higher aptitude students often performed sig- 
nificantly better than the lower aptitude students, both in 
the short and long term. 
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TABLE 3 



SUMMARY OF ANALYSES OF VARIANCE RESULTS 
RELATING 18 DEPENDENT VARIABLES 
TO TRAINING TREATMENT AND SCIENCE/MATH APTITUDE LEVELS 



DEPENDENT VARIABLES 


"BEST" 


VALUES OF 


F 


No. 


Description 


TREATMENT 


Treatment 


Aptitude 


Interacts 


1 


Time to Complete Training Course 


PTI 


26.38** 


2.00 


0.00 


2 


No. Interactions with Proctor 


PTI 


2.23 


1.12 


0.59 


3 


Pre-Course Knowledge 


CII 


3.42 


8.05** 


0.55 


4. 


Short-Term Knowledge 


• 


0.43 


72.43** 


1.19 


5 


Short-Term Knowledge Gain (4-3) 




0.00 


54.26** 


1.90 


6 


Long-Term Knowledge 




0.27 


44.20** 


1.24 


7 


Long-Term Knowledge Gain (6-3) 




0.05 


28.36** 


1.85 


8 


Forgetting (4-6) 




0.13 


2.51 


0.26 


9 


Short-Term Skill Test: 
Time to Complete All Items 




0.31 


21.45** 


1.66 


10 


No. T1med-0ut Items 




0.56 


10.90** 


0.72 


11 


No. Errors on Regular Items 




0.16 


6.38* 


1.47 


12 


No. Errors on Transfer Items 




0.00 


2.65 


0.44 


13 


Total Errors 




0.44 


0.40 


1.21 


14 


Long-Term Skil" Test: 
Time to Complete All Items 


Low H'igh 
PTI, CII 


1.29 


22.08** 


5.63* 


15 


No. Timed-Out Items 


Low High 
PTI, CII 


0.66 


14.58** 


4.80* 


16 


Ho. Errors on Regular Items 




0.00 


16.00** 


0.35 


17 


M5 .•••■Errors on Transfer Items 




0.90 


5.G4* 


0.00 


18 


Total Errors 




0.03 


18.64** 


0.30 



* p < .05 

** p 7 .01 
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Table 3 shows five analyses that suggest meaningful- 
treatment effects or aptitude-treatment interaction effects. 
Graphs of these effects were constructed to aid the interpre- 
tation of the results (Figures 22 through 26). The performance 
scales in these figures have been normalized across the 84 
subjects and scaled to have a mean of 500 and a standard de- 
viation of 10. Data had been scaled in this manner for the 
discriminant analysis programs. 

The PTI students finished the training in significantly 
less time than did the CI I students. Figure 22 shows approxi- 
mately a 1 standard deviation difference between the training 
treatments at both aptitude levels. This difference was 
significant well beyond the .01 level. 
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Figure 22. Time to Complete Training as a 
Function of Science/Math Aptitude and Training 
Treatment. 



The PTI students initiated fewer interactions with the 
proctor. Figure 23 shows both low and high aptitude PTI stu- 
dents engaged in fewer than the average number of interactions 
with the proctor during the course. The low aptitude CII 
students had more than the average number of interactions, and 
the high aptitude CII students were at about the mean. Neither 
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the main effects nor the interactions were statistically 
significant. However! these results are in accordance with 
the results of the analysis of the time to complete the course. 
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Figure 23. Number of Interactions as a Function 
of Science/Math Aptitude and Training Treatment. 

The above results are even more interesting in light of 
the results on the Pre-Course Knowledge Test. We had randomly 
distributed students who were matched on GCT'and ETST to the 
two training treatments. However, Figure 24 rhows that the 
CII students generally scored better on the Pre-Course Knowledge 
Test than did the PTI students. Yet he PTI students finished 
the course in less time and with fewer interactions with the 
training proctor than did the CII students. The treatment 
effect on the Pre-Course Kn n^.edgc Test approached but did not 
meet statist ical significance at the .05 level. 

Two analyses revealed an aptitude/treatment interaction. 
Figures 25 and 26 show that low aptitude PTi students took less 
time to complete the Long-Term Skill Test and had fewer items 
terminated because of the time limit than did the low aptitude 
CII students. To a lesser extent, the reverse was true for 
high aptitude CII and PTI students. The low aptitude CII 
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Figure 24. Pre-Course Knowledge as a Function 
of Science/Math Aptitude and Training Treatment. 

students were about 1 standard deviation "worse" than average 

on both of these criteria. All three of the other groups were 

at the mean or "better." In both cases, the interactions were 
significant at the .05 level. 
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Figure 25. Time To Complete 
the Long-Term Skill Test as 
a Function of Science/Math 
Aptitude and Training Treat- 
ment. 



Figure 26. Number tf Timed- 
Out Items on the Long-Term 
Skill Test as a function of 
Science/Math Aptitude and 
Training Treatment. 
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The difference between the low aptitude CII and PTI groups, 
on Time to Complete the Long-Term Skill Test was subjected 
to a t Test. The difference was significant at the .OS level. 
Because the number of timed-out items is logically correlated 
with the time to complete a test, we did not run a t Test on 
the low aptitude group difference shown in Figure 26. In 
reality, Figures 25 and 26 show two ways of investigating a 
single underlying phenomenon. 

Before leaving the univariate analyses, we want to dis- 
cuss two other results. 

The univariate tests to this point have dealt with 
science /math aptitude, as measured by the ETST. Recall that 
Figure 15, in the preliminary analysis, showed a small but 
consistent superiority of the lower verbal aptitude PTI stu- 
dents over the lower verbal aptitude CII students in time to 
complete the Long-Term Skill Test. We prepared these data for 
analysis of variance by defining high and low verbal aptitude 
groups in the same manner as we had for science/math aptitude 
in the discriminant analyses and analyses of variance reported 
above . 

Figure 27 shows a graph of the cell means for this analy- 
sis. The curves here resemble those in Figures 23, 25, and 26: 
The low aptitude CII group appears to be the "worst" performer. 
However, analysis of variance showed that neither the treatment 
nor interaction effects were statistically significant but the 
verbal aptitude effect was. 

It will be recalled that three problems from the Short-Term 
Skill Test were embedded in the Long-Term Skill Test. Students 
did not have access to the textbooks nor opportunity to practice 
troubleshooting digital logic circuits during the long-term 
retention interval. Furthermore, students were instructed not 
to d.tscusS the course with anyone; anecdotal evidence indicated 
that they followed those instructions to an extreme. 
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Figure 27. Time to Complete the Long-Term Skill 
Test as a Function of Verbal Aptitude and Training 
Treatment. 

Group performance on the three identical items for the 
Short-Term and Long-Term Skill Tests was compared in order to 
evaluate possible differences in forgetting troubleshooting 
skill. In general, improvement, rather than forgetting, was 
evident. Students completed the three items in less time after 
the long-term retention interval than they did immediately 
after completing the course. There was no clear pattern of 
differences for different levels of the aptitude or training 
treatment variables. 
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VI. Conclusions and recommendations 



We began this study by developing a relatively low-cost 
computerized instructional system to explore the feasibility 
of using such a system to alleviate the practical training 
problems in organizations plagued by limited funds and shortages 
of qualified instructors for highly varied curricula. We com- 
pared the computerized curriculum with a type of widely used 
non-computerized curriculum, the programmed text; we were very 
careful to make the comparison objective. 

We discovered that high aptitude students were superior 
to the low aptitude students on all the post-course perfor- 
mance criteria, whether the criteria were measures of acquired 
knowledge or task performance. The training treatment had no 
differential effect on the performance of high aptitude stu- 
dents. Low aptitude students in the non-computer curriculum 
appeared to perform better on certain skill criteria than did 
low aptitude students in the computerized curriculum. Students 
in the non-computerized curriculum finished the training 
sooner than those in the computerized curriculum, regardless 
of aptitude level. 

The results of the present study indicate that computerized 
instruction is not the optimum approach to at least some kinds 
of practical training problems. The computerized curriculum in 
the present study may be criticized as not representing the most 
elegant pedagogical use of a computer; however, this computerized 
curriculum doeo represent the main features of much of current 
computerized instruction. Insofar as much current CM is, in 
reality, an automated teaching machine, or an automatic programmed 
text, the present CII curriculum is representative of that ap- 
proach, even though we used a hard-copy text instead of the 
ubiquitous CRT to present textual information. Our approach 
was even a little more sophisticated than an automated teaching 
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machine, because we used the computer to drive a laboratory 
simulation of the operational equipment of interest. Further- 
more, we accomplished this with a minicomputer. 

We spent considerable effort to develop a computer- 
controlled training environment that would virtually replace 
the teacher and the laboratory needed for training a relatively 
complex skill, troubleshooting digital logic circuits. In meet- 
ing the needs for appropriate control conditions, we rejected 
the typically used classroorc-with- lecturer. We chose instead 
a curriculum that would benefit from the same thinking and 
organization as would the computerized curriculum, but would 
not require a computer. Our choice was programmed instruction. 

This last step, the choice of an appropriate control, 
distinguishes the present study from many others that have 
compared "computerized" with "conventional" instruction. The 
results of the present study demonstrate why this step is 
crucially important. Our results reinforce the need for con- 
tinued research and development of a sound theory and practice 
of instructional psychology. 

It would be inappropriate, of course, to generalize the 
present results to all training problems. Many variables 
distinguish specific training problems, and our study addressed 
only a few. From our results and review of certain relevant 
literature, however, we suggest the following principles in- 
volving the training variables of the present study: 

1. Barring the influence of relatively bizarre 
motivation, anxiety, and certain other vari- 
ables, high aptitude students will usually 
do well under any reasonable training condi- 
tions. This generalization may be tempered 
by the observation of Bialek ct al. (1973) 
that high aptitude military trainees, who 
are in classes that are aimed at the low ap- 
titude trainees, may become "bored and rest- 
less," and may show a loss in achievement . 



90 



2. Lower aptitude students are very sensitive 
to training treatment. They may require 
realistic simulations of operational equip- 
ment (McFann, 1971); they may do poorly 
using CRT-presented material on mathematical 
concepts (Dick and Latta, 1969); and they 
may learn certain skills as well or better 
from a programmed text as from interaction 
with a computer-controlled laboratory 
(present study). Much additional research 
appears to be necessary before a set of 
general principles for training lower apti- 
tude students can be formulated. 

3. The computerized instructional approach may 
not always be the fastest means of training. 
While one CM curriculum resulted in a large 
time savings compared to lecture/classroom 
training (Ford et al. , 1972), our computer- 
ized training was significantly elower than 
an equivalent programmed-text curriculum. 

4. The computerized approach may require, at 

least at the initial stage of training, •* 
more proctor intervention and help than 
some other techniques. 

5. If a computerized approach is called for in 
some training application, it may not be as 
expensive to implement as have the CAI sys- 
tems of the past. Recent workers (Thomas, 
1973; Brebner, 1973; Mecherikoff, 1974; the 
present study) have shown progress in imple- 
menting computerized instruction on relatively 
small, inexpensive systems. Also, there are 
continuing developments in expanding the num- 
ber of terminals serviced by big systems so 
that the cost-per-hour of user time is re- 
duced. The growing success of commercial 
time-sharing systems (e.g., General Electric's 
Mark III system) is evidence for hope along 
the latter lines. 

While there may be hope for the future of individualized 
training, there is a great deal of work in this area yet to be 
done. Research in this area is time consuming and expensive. 
Mickey and Newton (1967) note four criteria for adequate re- 
search in training comparisons: The study should (1) involve 
at least five hours of instruction, (2) use more than 15 



students in each group, (5) report the time to complete the 
training, and (4) report pre- and post-training results. The 
first criterion ensures that the study concerns training and 
curriculum problems as opposed to laboratory learning phenomena 
(e.g., paired-associate learning). The second criterion ensures 
the reliability of the results. The last two criteria ensure 
that at least the basic dependent variables are measured and re- 
lated to students' pre-course abilities. The present study met 
all these criteria and at least one other most important one: 
equivalence of experimental and control curricula content. 

Meeting those criteria, unfortunately, requires much 
larger research studies than those typically reported in the 
literature on human learning. As Locke (197)) stated, private 
industry is not keen on solving these problems with internal 
funds. Evidently, government agencies with a vested interest 
in this area are faced with the prospect of continuing to 
support training research, or accepting seat-of-the-pants 
training strategies that may be based on little more than 
(costly) fads. The results of the pvesent study lead to the 
following recommendations: 

1. Research in computerized instruction mast in- 
clude adequate control curricula for compari- 
sons. Furthermore, such studies should devise 
means of realistically assessing the average 
amount of instructor time per student in the 
experimental and control conditions. Many 
research reports on computerised instruction 
do not cite what is often admitted in private, 
that a higher instructor-to-student ratio is 
present in the computerized curriculum, which 
was originally intended to replace the instruc- 
tor, than in the "conventional" curriculum. 

2. Research on optimizing instruction for indi- 
viduals of varying aptitudes should focus on 
the lower aptitude students. It would be an 
overstatement to say that high aptitude stu- 
dents will take care of themselves in all 
cases; however, the results of the present 
study and those cited earlier suggest that 
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different training strategies do not usually 
result in large performance differences among 
high aptitude students. On the other hand, 
there is adequate evidence of significant 
performance differences among low aptitude 
students who are subjected to different train- 
ing treatments. Research in this area should 
continue. It should be noted that "low apti- 
tude" is a relative term. In the present 
study, we dealt with students whose math/ 
science aptitude was low relative to the 
sophistication of the digital logic training 
material. Probably, their math/science apti- 
tude, and certainly their general verbal ap- 
titude, would be considered moderate relative 
to training in infantry skills in the studies 
by Bialek discussed earlier. 

The instructional practitioner who cannot wait 
for the results of further research relevant 
to' his training area should carefully consider 
relatively inexpensive alternatives, like high- 
quality programmed instruction, before trading 
in his classrooms and lecturers for computer- 
ized instruction. The course reconstruction 
and reformatting that his staff must perform 
before beginning to implement a computerized 
curriculum would constitute the bulk of the 
work and expense necessary for a high-quality 
programmed instruction curriculum. 
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APPENDIX A 

Excerpt from Chapter 5, "R-S Flip-Flop," 
in the PTI textbook 

A Programmed Text in DIGITAL TECHNOLOGY 
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APPENDIX B 

Excerpt from Chapter 5, "R-S Flip-Flop, " 
in the CI I textbook 
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APPENDIX C 
Knowledge Test 



Name 
Number 



DIGITAL TECHNOLOGY 
FIN4L 

Do not open this test booklet until your instructor tells 
you to. 

This test consists of 40 multiple -choice questions. In 
the questions that follow, choose the single response that is 
most correct. Read each question completely and carefully 
before answering. 

NOTE: Sometimes you may read the answer *and immediately decide 
that one answer is correct. Do not stop there. It may be that 
more than one answer is correct, and the last alternative answer 
may indicate multiple-correct answers. 



1. In a three-branch parallel switching circuit, a signal 
will pass from one side of the circuit to the other: 

a. When all three switches are closed. 

b. When any single switch is closed. 

c. When all three switches are open. 

d. Both (a) and (b) are correct. 

2. In a negative logic system: 

a. A 1 will be represented by a +1 volt, and a zero 
will be represented by 0 volts. 

b. If a 1 were represented by -5 volts, a 0 might be 
represented by -10 volts. 

c. If a 1 were represented by 0 volts, a 0 might be 
represented by -1 volt. 

d. If a 1 were represented by -2 volts, a 0 might be 
represented by +2 volts. 

3. Which of the following statements about the AND gate is 
incorrect? 

a. It always has one output. 

b. It always has two or more inputs. 

c. It always introduces a time delay. 

d. An active level always occurs at the output when 
there are active levels on all inputs simultaneously. 



c - 4 

ERIC 



4. Refer to the table, 
should be: 

a. 0 0. 

b. 0 1. 

c. 1 0. 

d. 1 1. 



The top and bottom values in this table 



0 
0 
0 



8 



0 
0 



0 
1 
0 
1 
0 
1 
0 
1 



B x . C 



? 
0 
0 
0 
0 
0 
0 

? 



5. Refer to the figure. The Boolean expression describing 
the logic function shown is: 

a. A ♦ B « C 

A 

b. X ♦ F ■ C. 

c. A ♦ B « E. _B 

d. X ♦ F ■ E, 




6. The Boolean expression for the NOR gate is: 

a. A ♦ B « C. 

b. A ♦ B « C. 
C. A x B ■ C 
d. A x B ♦ E 

7. Refer to the table. The Boolean expression that should 
appear at the top of the right-hand column is: 

a. X ♦ B. Input Output 

b. A ♦ F. A B ? 
C. A x B. 
d. X x F. 



0 

0 

1 
1 



0 
1 
0 

1 



1 

0 
0 
0 
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8. An OR gate can be constructed of 

a. an AND gate with inverted input. 

b. an AND gate with inverted inputs and output. 

c. an OR gate with inverted inputs and output. 

d. a NOR gate with inverted output. 

9. The difference between the logic functions that have the 
Boolean expressions X ♦ F » C and A ♦ B • T? is that: 

a. The first has inverters on the inputs, the second 

« 

on the output. 

b. The first has inverters on the output, the second 

on the inputs. 

c. They both have inverters in the same locations, 
but the expressions have been stated differently. 

d. One is constructed of an OR gate, the other of an 
AND gate. 

10. Refer to the table. The Boolean expression that should 
appear at the top of the right-hand column is: 

a. X x B. A B ? 



b. 7TTX 



0 0 1 
Oil 

C. A ♦ B. 10 1 

d. inrs. iio 
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11. A logic function which satisfies the Boolean expression 
A x (B + C) consists of 



a. a two- input OR gate connected to one input of a 
two- input AND gate. 

b. a two- input AND gate connected to one input of a 
two -input OR gate. 

c. two AND gates with their outputs connected to 
an OR gate. 

d. a three-input OR gate connected to one input of 
a two -input AND gate. 



12. If a simple switching circuit is represented by the Boolean 
expression (A ♦ B) x C, 

a. A and B and C must all be closed to close the circuit. 

b. closing A and B will close the circuit. 

c. closing A and C will close the circuit. 

d. Both "a" and "b" are correct. 

13. Refer to the figure. The Boolean expression describing the 



logic function shown is: 
a. OC ♦ B) x C - ff. 



b.. a ♦ cnm) - IT. 



c. (A + B) x C « U. 




d. (A ♦ F) x C « D. 
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14. Refer to the figure. The Boolean expression describing 
the logic function shown is: 

a. (A x C) ♦ (B x D) - E. 

b. (A x B) ♦ (C x D) - E. 

c. (A x B) x (C ♦ D) * E. 

d. (A x B) ♦ (A x C) ♦ (A x D) MB x C) ♦ (B x D) - E. 




E 



IS. Refer to the figure. The Boolean expression describing 
the logic function shown is: 

a. (A x B) ♦ (B x C) ■ D. 

b. (A ♦ B) ♦ C ■ D. 

c. (A x B) ♦ C « D. 

d. (A ♦ B) x C - D, 




D 
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16. Logic devices with memory 

a. are similar to logic devices without memory 
because their output condition depends solely 
upon their input condition. 

b* can be constructed of logic devices without memory. 

c. work like an automobile horn. 

d. are too large to be used in computers. 

17. An unclocked R-S flip-flop, when presented with two Os, 

a. reverses state. 

b. goes to the 0 state. 

c. goes to the 1 state. 

d. retains its initial state. 

18. The R-S, J-K, and D flip-flops have in common that: 

a. they retain a state even after their inputs have 
been removed. 

b. their output condition depends upon input conditions 
at a particular point in time. 

c. their output condition depends upon input conditions 
in the past. 

d. All of the above are correct. 



9 
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19. Clocks are most useful for 

a. changing the states of flip-flops. 

b. controlling the precise time at which a change of 
state of a flip-flop occurs. 

♦ 

c. establishing the initial state of a flip- Top so that 
it is "enabled" for triggering. 

d. keeping the flip-flop from "running away." 

20. One of the problems with both clocked and non- clocked R-S 
flip-flops is that: 

a. they both have an indeterminate state. 

b. the precise time at which they change state cannot 
be controlled with accuracy. 

c. they retain their initial state if presented with 
two "1" pulses. 

d. they reverse their initial state if presented with 
two "0" pulses. 

21. The D flip-flop 

a. has four states. 

b. has only two states. 

c. has two determinate states and two indeterminate 
states. 

d. has four determinate states. 
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22. If a D flip-flop receives a "0" pulse, the levels at the "1" 
and "0" output terminals after clocking will be, in order: 

a. 0 0. 

b. 0 1. 

c. 1 0. 

d. 1 1. 

23. The J-K flip-flop 

a. has the same truth table as the (unclocked) R-5 
flip-flop. 

b. has the same truth table as the D flip-flop. 

c. has the same truth table as the clocked R-S 
flip-flop. 

d. has the same truth table as the D flip-flop, 
except for the indeterminate and complementing 
states. 

24. If a J-K flip-flop is in the "0" state, and a 1 is placed 
on the J terminal and a 0 on the K terminal, 

a. it will go to the "1" state at the start of the 
clock pulse. 

b. it will go to the "0" state at the start of the 
, clock pulse. 

c. it will go to the "1" state at the end of the 
clock pulse. 

d. it will go to the "0" state at the end of the 
clock pulse. 

C-ll 
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25. When two Os are presented on the input terminals of a 
J-K flip-flop, and the device is clocked, 

a. it retains its initial state. 

b. it goes to the 0 state. 

c. it goes to the 1 state. 

d. it complements its initial state. 

26. A four-bit shift register 

a. consists of four flip-flops. 

b. consists of a circuit with four storage positions, 
though these do not necessarily correspond to the 
number of flip-flops. 

c. can only be constructed of J-K flip-flops. 

d. may consist of one four-bit flip-flop. 

27. If a four-bit shift register, consisting of four stages, 
A, B, C, and D, initially contains the binary number 1101 , 
and then the input is put in the 0 state, after three 
clock pulses C and D will contain: 

a. 00 

b. 01 

c. 10 

d. 11 



o 
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28. If a sixty-seven-bit shift register contains 67 Is, it will 
take 

a. 133 clock pulses to clear the register and fill 
it with Os. 

b. 134 clock pulses to clear the register and fill 
it with Os. 

c. 68 clock pulses to clear the register. 

d. 67 clock pulses to clear the register. 

29. Zeroes and ones shift between stages of a J-K flip-flop 
shift register 

a. as soon as the clock pulse occurs. 

b. as soon as the clock pulse terminates. 

c. as soon as the input pulse is received. 

d. as soon as the input pulse and clock pulse occur. 

30. A ring counter 

a. is a divider with the output connected back to 

the input. 

b. is a divider with the output inverted and connected 
back to the input. 

c. is a shift register with the output inverted and 
connected back to the input. 

d. is a shift register with the output connected 
back to the input. 
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31. If a four-stage Ting counter, with stages A, 8, C, and D, 
initially holds the binary number 1101, its state after 
three clock pulses will be 

a. 1110 

b. 0111 

c. 1011 

d. 1101 

32. If a four-stage switch- tail ring counter, with stages 
A, B, C, and D, initially holds the binary number 1101, 
its state after three clock pulses will be: 

a. 0110 

b. 1011 

c. 0010 

d. 0101 

33. If a four-stage switch-tail ring counter initially contains 
the binary number 1111, 

a. in four clock pulses it will contain the binary 
number 1111. 

b. in four clock pulses it will contain the binary 
number 0000. 

c. in eight clock pulses it will contain the binary 
number 0001. 

d. in eight clock pulses it will contain the binary 
number 1110. 
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changes at the same rate as the input of the first 
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34. A divide-by-sixteen divider, consisting of R-S flip-flops, 

a. would have two stages. 

b. would have three stages. 

c. would have four stages. 

d. could not be constructed, because dividers must con- 
tain J-K flip-flops. 

35. In a divide-by-sixteen divider, the output of the first 
flip-flop 

flip : flop. 

b. changes at one -fourth the rate of the inpuc of the 
first flip-flop. 

c. changes at half the rate of the jinput of the first 
flip-flop. 

d. changes at twice the rate of the input of the first 
flip-flop. 

36. The J-K terminals of a divide-by-sixteen divider » 

a. are eliminated. 

b. are tied together and connected to the clock. 

c. are tied together and continuously enabled by . 
a 1 level. 

d. are tied together and grounded. 

37. A divide-by-32 divider would contain 

a. 4 stages. 

b. S stages. 

c. 6 stages. 

d. 7 stages. 

C-15 



38. A multiplexer 

a. switches information from several inputs to a 
single output. 

b. switches information from a single input to 
several outputs. 

c. stores information from several inputs before 
switching it to an output. 

d. delays information between input and output. 

39. A ring counter is used in a multiplexer to: 

a. switch each input channel, in turn, to the 
output channel. 

b. switch each output channel, in turn, to the 
input channel. 

c. recirculate the input pulses for one cycle 
before switching them to the output. 

d. circulate the input through each AND gate 
before switching it to the output. 
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40. Refer to the figure. The levels on points A, B, C, and 
D are 1, 0, 1, 0, and a 1 is circulating in the ring 
counter, beginning initially at A'. The output at point 
E will read, across time, 

a. 0000. 

b. 0101. 

c. 1010. 

d. 1111. 
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APPENDIX D 



Annotated Instructions To Students 
for the Skill Tests 



PORN A: SHORT-TERM SKILL TEST 



SUPER LOGIC PROBLEMS 



Switches 



A. You have up to 5 minutes to explore Logic Block #1. Use the 
Grey Box to put i's and 0's to the upper and lower inputs of 
this logic block. Use the probe to explore what happens to 
the outputs of the different components within Logic Block 

• 1. ® 



[If necessary, help the student wire up] 
this example. J 



Okay, time is up--we will go on to a troubleshooting problem. 
^Disconnect LB #1 and connect LB #2?] 

B. There is one fault or error in Logic Block #2. Find it as 
quickly as you can and describe it to me. 

(Gate 1 is bad; it behaves as an OR gate instead] 
[of an AND gate. J 

[Disconnect LB #2 and connect LB #3?] 

C. There is one fault in Logic Block #3. Find it as quickly as 
you can and describe it to me. 

[Faulty output of Gate 3— OUTPUT REMAINS HIGH 
or may be interpreted as an input from Gate 4 
pulling output of Gate 3 HIGH. 

[disconnect LB #3 and connect LB #4^ 

D. There is one fault in Logic Block #4. Find it as quickly as 
you can and describe it to me. 

[Faulty input of Gate 4— INPUT REMAINS LOW and"] 
OUTPUT REMAINS HIGH. J 
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FORM A: SHORT-TERN SKILL TEST 



HALF ADDER PROBLEMS 



Switches 



You hav* learned about shift registers and how to enter infor- 
mation into them. There are three numbers on this [jB ive t0 
student] Scratch Sheet. In a moment, I want you to enter the 
first number into the first shift register. I will connect 
the wires to the first shift register [connect]. We press 
the clock button in order to get data into the register. 
^Enter one 1, then 0*s to clear register!] 

1* Okay, enter the first number on the Scratch 
Sheet into the shift register. (?) 

2. Okay, set up the wires to the second shift 
register and enter the second number from 
the Scratch Sheet. @ 

3. Okay, enter the third number into the third 
shift register. (?) 

[Remove all wires 7] 

Now, set up the inputs to Logic Block #1 and determine the 
truth table for its inputs and output, as shown on the 
Scratch Sheet. [Give Subject a pencil]] (7) 




Correct 
Answer 



I will set up this circuit so that the two shift registers 
input into the logic block and the logic block outputs to the 
third shift register. You will have up to 8 minutes to explore 
this circuit. £connect--Enter 0»s to clear all registers!] 
Now, don't forget the truth table you have just determined for 
the logic block. Go ahead and manipulate this circuit. (?) 



D. 



[^Everything should bo cleared at this point.J 

Now I am going to introduce a fault into the circuit, 
it as quickly as you can and describe it to me. 



Find 



Change 
Switches 



© 



[In Shift Register #3, the output of FF #< 
remains HIGH. 

E. I will fix that fault and introduce a new one; fine it and 
describe it to me. 



Change 
Switches 



© 



Faulty output of Gate 1 in the logic block — OUTPUT 
REMAINS LOW may be interpreted as faulty input of 
Gate 3 pulling output of Gate 1 LOW. 
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FORM A: SHORT-TERM SKILL TEST 



MULTIPLEXER PROBLEMS 



Switches 



A. 



B. 



C. 



D. 



E. 



The shift register in this circuit has a switch on the input 
so that you can enter information while the switch is in the 
down position and then circulate the information by putting 
the switch in the up position. Set up the Cloak and Data Input 
wires to get a single 1 to circulate in this Ring Counter. (?) 



[Be s 
Subj 



ure switch is in down position before 
ect starts. 



0 



Now clear the Ring Counter, disconnect all the wires, and lay 
them on the table. 

Now set up 4 data input wires so that you can enter informa- 
tion into the upper inputs of the 4 AND Gates. Set the inputs 
to 1011 from top to bottom as on your scratch sheet and verify 
it with the probe. Do not use the Ring Counter at all for 
this exercise. 

Now get a single 1 circulating in the Ring Counter again. 
Observe the output of Logic Block #2 as you clock the 1 
around the Ring Counter. Explore the circuit with the probe 
for a few minutes so that you understand it. @ 



[Have 1011 in 
at beginning 



data input and a 1 in Ring Count 
of next test. 



I will now introduce a fault in the circuit, 
describe it to me. 



Find it and 



Change 
Switches 



Faulty output of Gate 1 in Logic Block #l--OUTPUT 
REMAINS HIGll--may be interpreted as input from 
Gate 1 of LOGIC BLOCK #2 pulling output of Gate 1 
high. 



ave 1011 in data input and a I in Ring Coun 
at beginning of next test. 



terj 



F. I will fix the firs,: fault and introduce another. Find it 
and describe it to me. 







Change 
Switches 





[Paul 
INPU' 



ty input of Gate 3 in Logic Block #2-- 
INPUT REMAINS LOW. 



3 



Okay, you're finished. Thank you for your participation. 
Please don't discuss this test with anyone else. We'll be 
seeing you in a few weeks again. 
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FORM B: LONG-TERM SKILL TEST 




SUPER LOGIC PROBLEMS 



A. 



B. 



C. 



D. 



Connect two input wires to Logic Block #1. There is one 
fault in this circuit. Find it; tell me the name and 
number of the faulty component; and describe in your own 
words what the problem is, (7) 



Lower input in Gate 4 remains HIGH so that all 

it takes is a high at the upper input to cause 

a high output. Should require 2 high inputs 
to make output of this AND Gate high. 



Disconnect Logic Block #1 and connect the inputs to Logic 
Block #2. The problem this time is, for you to determine 
if Logic Block #2 is working properly. When you have 
reached a decision tell me "Yes, it's working properly/' 
or "No, it's not working properly." (7) 



YES, it is working properly. 



Disconnect Logic Block #2 and connect Logic Block *3. 
There is one fault in this circuit. Find it; tell me 
the name and number of tne faulty component and describe 
the problem. (7) 



Output of Gate 3 remains high for all input 
conditions. This AND Gate should require 2 
high inputs for a high output. 



Disconnect Logic Block #3 and connect Logic Block #4. 
There is one fault in this circuit. Find it; tell me 
its name and number; and describe the problem. (7) 



Upper input in Gate 4 remains HIGH so that all 
it takes is a high at the lower input to cause 
a low output. Should require 2 high inputs to 
make the output of this NAND Gate low. 
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FORM B: LONG-TERM SKILL TEST 



A. 




HALF ADDER PROBLEMS 
(Clear All Shift Registers) 



1. Connect an Input wire and a Clock wire to Shift 
Register #1. Enter the first number on this 
scratch sh< et (hand S scratch sheet). The number 
is 0001. @ 

2. Enter the second number in Shift Register #2. 
The number is 1001. © 

3. Enter the third number in Shift Register #3. 
The number is 1011. Q) 

B. I'll connect up the entire circuit now (connect and clear 
all registers) . Now I am going to introduce a fault into 
the circuit. Find it; tell me the name and number of the 
faulty component; and describe in your own words what the 
problem is. 




Output of Flip-flop 3 in Shift Register #2 
remains LOW; this also means it won't pass 
a 1 on to Flip-flop 4. 



C. I'll fix that fault and introduce a new one. Find it; 
tell me its name and number; and describe the problem 
(Clear All Shift Registers). 



Change 
Switches 



0 



Output of Flip-flop 2 in Shift Register #1 
remains HIGH; this also means it keeps pass- 
ing l»s on to the rest of the shift register 
and it can't be cleared. 



I'll fix that fault and introduce a new one. Find it; 
tell me its name and number; and describe the problem 
(Clear All Shift Registers). 




Output of Gate 1 in the Logic Block remains 
LOW. This OR Gate should have a high output 
if either input is high. 
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FORM B: LONG-TERM SKIL'u TEST 




MULTIPLEXER PROBLEMS 
(Clear Ring Counter) 



A. Data can be entered into the Ring Counter when the switch 
is in the down position; and data can be circulated when 
the switch is up (leave switch down). Set up the Ring 
Counter to circulate a single 1. ^F) 

B. I'll set the data inputs to the 4 AND Gates to 1010 (set 
up; and get 1 into first position in Ring Counter); and 
I'll introduce a fault into the circuit. 




When you are looking for the fault, you may enter new 
data into the AND Gates or Ring Counter if you want to. 
Find the fault; tell me its name and number; and 
describe the problem. 



Flip-flop 3 in the Ring Counter remains LOW; 
it won't pass the one on to Flip-flop 4. 



C. I'll fix that fault and introduce a new one. First 
set the data inputs to 1011 and get a 1 in the Ring 
Counter (set data and Ring Counter) . 



I'll 




Find the fault; tell me its name and number; and 
describe the problem. Q) 



Output of Gate 1 in Logic Block #1 remains 
HIGH. This AND Gate should have a high 
output only when both inputs are high. 



D-ll 



0. I'll fix that fault and introduce another; find it; tell 
me its name and number; and describe the problem. 




Output of Gate 3 in Logic Block #2 remains 
LOW. This OR Gate should have a high out- 
put when either input is high. 
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